diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index 01539b45..985df013 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -17,7 +17,7 @@ Then perform these steps in order: 1. Run `git checkout -b release/NEW_VERSION` 2. Replace `OLD_VERSION` with `NEW_VERSION` in all of the following files: - `pyproject.toml` — the `version` property - - `ably/__init__.py` — lib_version value + - `ably/pubsub/version.py` — lib_version value 3.Run `uv sync` to update `uv.lock` file 4. Commit all files together with message: `chore: bump version to NEW_VERSION` 3. Fetch merged PRs since the last release tag using: diff --git a/.claude/skills/uts-to-python/SKILL.md b/.claude/skills/uts-to-python/SKILL.md index a413d6bb..48f18fc9 100644 --- a/.claude/skills/uts-to-python/SKILL.md +++ b/.claude/skills/uts-to-python/SKILL.md @@ -20,6 +20,9 @@ gh api 'repos/ably/specification/contents/uts/realtime/integration/.md' -- gh api repos/ably/specification/contents/uts/docs/proxy.md --jq '.content' | base64 -d gh api 'repos/ably/specification/contents/uts/rest/integration/proxy/.md' --jq '.content' | base64 -d gh api 'repos/ably/specification/contents/uts/realtime/integration/proxy/.md' --jq '.content' | base64 -d +gh api 'repos/ably/specification/contents/uts/objects/unit/.md' --jq '.content' | base64 -d +gh api 'repos/ably/specification/contents/uts/objects/integration/.md' --jq '.content' | base64 -d +gh api repos/ably/specification/contents/uts/objects/helpers/standard_test_pool.md --jq '.content' | base64 -d ``` The realtime integration tier nests, so list a directory before fetching from it: @@ -39,9 +42,10 @@ carries over as it stands: `channels/channel_publish_test.md` becomes `channels/channel_publish_test.py`. Every directory needs an `__init__.py`, as `test` is a package. -There are two kinds of tier. `rest/unit` and `realtime/unit` serve every request from a -mock and reach no network; `rest/integration` and `realtime/integration` run against the -real Ably sandbox and have no mock at all. See **The integration tier** below. +There are two kinds of tier. `rest/unit`, `realtime/unit` and `objects/unit` reach no +network; `rest/integration`, `realtime/integration` and `objects/integration` run against +the real Ably sandbox and have no mock at all. See **The integration tier** below, and +**The objects tier** for LiveObjects. `test/uts/rest/unit/time_test.py` is the reference example for REST unit, `test/uts/realtime/unit/connection/auto_connect_test.py` for realtime unit, @@ -451,6 +455,105 @@ async def test_rsc15l4_cloudfront_header_fallback(sandbox, proxy_session): assert http_responses(log)[0]['status'] == 403 ``` +## The objects tier + +`uts/objects/unit/.md` becomes `test/uts/objects/unit/_test.py`, the integration +specifications keep their `_test` suffix under `test/uts/objects/integration/`, and +`integration/proxy/objects_faults.md` becomes +`test/uts/objects/integration/proxy/objects_faults_test.py`. Seven unit specifications are +pure and construct internal objects; the other eight drive `channel.object` over the mock +websocket. Follow `internal_live_map_test.py` (pure), `path_object_mutations_test.py` (mock) +and `objects_lifecycle_test.py` (integration). The integration package's own `conftest.py` +provisions a separate app under the realtime tier's fixture name, `realtime_sandbox`, and +all three of its specifications take `use_binary_protocol`. The package is +`ably.pubsub.liveobjects`, named for the product as PDR-080 has package names do, while +`test/uts/objects/` keeps the specification's directory name so that each test file maps +onto its specification; the public names are exported from `ably.pubsub.server.liveobjects`. + +**Translate the untyped API through the typed views** (LODR-061, the RTTS partition). The +specifications call every method on one `PathObject` or `Instance` class; ably-python +reaches a type's methods through its view: + +| Pseudocode | ably-python | +|---|---| +| `root.get("score").value()` | `root.get('score').as_live_counter().value()` | +| `root.get("name").value()` | `root.get('name').as_primitive().value()` | +| `pathObject.set("k", v)` | `await path_object.as_live_map().set('k', v)`; `root` is already a `LiveMapPathObject` | +| untyped `value() == null` | `as_primitive().value() is None` **and** `as_live_counter().value() is None` | +| `inst.id()` | `inst.id`, a property, as is `inst.type` | +| `{ depth: n }` | `subscribe(listener, depth=n)`, keyword-only | + +Mutations are `async`; reads, navigation, views and `subscribe` are not. `entries()` yields +tuples. A path's views never raise, and a write through the wrong one raises 92007, or 92005 +for a path that does not resolve; an `Instance`'s views raise 92007. A spelling difference is +not a deviation. Where the partition makes a read unreachable — `value()` on a +`LiveMapInstance` — assert the `type`, `not hasattr(...)` and the 92007, and cite S-5. + +**The pure tier adapts to five shapes**, defined in `deviations.md` and cited by label in the +module docstring: S-1 `apply_operation` returns a boolean, so record updates with +`capture_updates(obj)`; S-2 the sync state machine is a standalone `RealtimeObject()`'s +(`_on_attached(has_objects)`, `_handle_object_sync_messages(msgs, channel_serial)`, +`_sync_state`); S-3 `evaluate(vt, timestamp_ms)`; S-4 the retained create is +`operation.resolved_counter_create` / `resolved_map_create`; S-5 above. Members of a public +class that LODR-061 does not name carry a leading underscore (`channel.object._objects_pool`). + +`test/uts/objects/helpers/standard_test_pool.py` holds what the specifications share: +`setup_synced_channel` (and `_no_ack`), `standard_mock_websocket`, `objects_client`, +`objects_channel_options`, `objects_connected_message`, `objects_attached_message`, the +`build_*` builders, `json_value`, `bytes_value`, `ack_serial`, `remote_serial`, +`below_ack_serial`, `object_message(s)`, `capture_updates`, `build_public_object_message`, +`assert_unchanged_after_quiescence`, `provision_objects_via_rest`, and the wire constants +(`HAS_OBJECTS`, `OBJECT_SUBSCRIBE_FLAG`, `OBJECT_PUBLISH_FLAG`, `LWW`, the actions). +`test/uts/README.md` says what each is. + +### Traps found while deriving and implementing the objects tier + +- **An injected frame is applied on the transport's read task, not by `send_to_client`.** + Read state only after a `poll_until` on its effect. A negative — "did not fire", "the echo + was not applied" — needs a positive control sent *behind* the message under test and + `assert_unchanged_after_quiescence`; an exact count after `poll_until(>= n)` needs + `await settle()` first; and a subscription made straight after a seeding message receives + the seed, so poll for the seed before subscribing. The specifications often skip all three, + and their negatives then pass whatever the SDK does. +- **Process a re-sync ATTACHED before starting what it should hold back.** Send it, poll until + `channel.object._sync_state == ObjectsSyncState.SYNCING`, then start `get()` or the write as + a task; started earlier, it sees SYNCED and resolves at once. +- **A mock ATTACHED grants no modes.** The standard one carries `HAS_OBJECTS` alone, which + empties `channel.modes`, and RTO2 then checks the modes the channel *requested* — so request + them with `objects_channel_options()` or `get()` raises 40024. Grant modes as `flags` bits, + never as the specifications' `modes: [...]`. An injected `flags: 128` at the integration tier + empties them the same way. +- **A write resolves on its ACK.** The standard mock ACKs each OBJECT with + `ack_serial(msgSerial, i)`; under `setup_synced_channel_no_ack`, drive the write as a task. + Those serials land in `appliedOnAckSerials`, so never reuse one as an inbound serial, and + use `remote_serial(n)` for a remote write: a bare `'99'` sorts before `POOL_SERIAL` and is + stale. +- **`json` values are JSON strings on the wire** (OD2g), so compare a captured + `mapSet.value.json` after `json.loads`; `bytes` are base64. Actions are + `ObjectOperationAction`, an `IntEnum`. +- **Numbers decode to `float`.** Read them with `value(float)`; `value(int)` raises + `TypeError`, since `expected` must be exactly one of `str, float, bool, bytes, list, dict`. + Assert booleans with `is True`, since `True == 1`. +- **Creating a `LiveCounter` or `LiveMap` reads `/time`.** `objects_client` answers it; a + client built any other way needs `mock_http=time_mock_http(clock)`. +- **A tombstoned object reads null with no GC at all**, so a GC test asserts that the object + left `_objects_pool`. The GC interval, 300000 ms, is read when the first ATTACHED schedules + the timer on the client's clock: set `channel.object._gc_interval_ms` before `get()`. +- **The public and internal `ObjectMessage` share a name.** Reach the public one as + `publicmessage.ObjectMessage`, from `ably.pubsub.liveobjects`. +- **Check that a test can fail.** Several objects specifications assert something that holds + with the behaviour removed. The derivation caught them by running each module against a + throwaway reference implementation with one fault injected; add the discriminating + assertion under `# UTS SPEC ERROR:` and record the fault. +- **At the sandbox, the echo arrives before the ACK**, so RTO9a3's dedup is only reachable at + the mock tier, and **ACKs are paced at about one per 500 ms per connection**, so each + awaited write after the first can take half a second. +- **Every objects channel gets an OBJECT_SYNC**, an empty one included, and a resumed + ATTACHED on an attached channel restarts the sync without a state change: wait on + `get()`, not on a channel state. A sync cursor can itself contain `:`. +- **uts-proxy matches `action` as a string and names actions only up to AUTH**: OBJECT_SYNC + is `'20'`, OBJECT `'19'`. Its `delay` holds every later frame behind the delayed one. + ## Traps that cost the most time Ordered by how much they cost, not by subject. Every one was hit for real while @@ -728,7 +831,7 @@ Established against the real proxy and the real sandbox while deriving asks for a token through a client pointed at the session puts a request in front of the waiting rule and an extra `http_request` in the log, which breaks every assertion that counts requests exactly (`== 1` for RSC15l's 4xx test, `>= 2` for the - fallback ones). Build an inner `AblyRest(key=api_key, endpoint=SANDBOX_ENDPOINT)`, + fallback ones). Build an inner `create_http_client(key=api_key, endpoint=SANDBOX_ENDPOINT)`, request the token through that, and close it. - **`http_response` events carry no `path`.** They have `status` and `ruleMatched` only, so "the injected response fired" is read off the response events **in order** @@ -896,8 +999,8 @@ the reasoning. The next reader will otherwise reach the same first conclusion. ```bash uv run --frozen --extra crypto --extra dev ruff check ably/ test/ -uv run --frozen --extra crypto --extra dev pytest test/uts/rest/unit test/uts/realtime/unit test/uts/helpers -q -uv run --frozen --extra crypto --extra dev pytest test/uts/rest/integration test/uts/realtime/integration -q +uv run --frozen --extra crypto --extra dev pytest test/uts/rest/unit test/uts/realtime/unit test/uts/objects/unit test/uts/helpers test/uts/objects/helpers -q +uv run --frozen --extra crypto --extra dev pytest test/uts/rest/integration test/uts/realtime/integration test/uts/objects/integration -q RUN_DEVIATIONS=1 uv run --frozen --extra crypto --extra dev pytest test/uts -q ``` @@ -906,7 +1009,7 @@ environment's cutoff — and `--extra dev` carries pytest. Line length is 115. I `uv.lock` changes, revert only what the run touched; a wholesale `git checkout -- uv.lock` takes any edit of your own with it. -The second command is the offline tiers, which need no network. The third is the two +The second command is the offline tiers, which need no network. The third is the three integration tiers, each of which provisions a sandbox app of its own and does need the network; `pytest test/uts -q` runs everything together, so run the tiers separately when only one is in question. All three must pass. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 2a7a2ae2..f40e27e9 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -42,27 +42,27 @@ jobs: with: name: python-package-distributions path: dist/ - - name: Check that wheel and tarball contains ably/sync/ + - name: Check that wheel and tarball contains ably/pubsub/sync/ run: | # Check wheel WHEEL=$(ls dist/*.whl | head -n 1) echo "Checking wheel: $WHEEL" - if unzip -l "$WHEEL" | grep -q "ably/sync/"; then - echo "✅ Found ably/sync/ in wheel" + if unzip -l "$WHEEL" | grep -q "ably/pubsub/sync/"; then + echo "✅ Found ably/pubsub/sync/ in wheel" else unzip -l "$WHEEL" - echo "❌ ably/sync/ not found in wheel" + echo "❌ ably/pubsub/sync/ not found in wheel" exit 1 fi # Check tarball TARBALL=$(ls dist/*.tar.gz | head -n 1) echo "Checking tarball: $TARBALL" - if tar -tzf "$TARBALL" | grep -q "ably/sync/"; then - echo "✅ Found ably/sync/ in tarball" + if tar -tzf "$TARBALL" | grep -q "ably/pubsub/sync/"; then + echo "✅ Found ably/pubsub/sync/ in tarball" else tar -tzf "$TARBALL" - echo "❌ ably/sync/ not found in tarball" + echo "❌ ably/pubsub/sync/ not found in tarball" exit 1 fi @@ -74,7 +74,7 @@ jobs: runs-on: ubuntu-latest environment: name: pypi - url: https://pypi.org/p/ably + url: https://pypi.org/p/ably-pubsub-server permissions: id-token: write # IMPORTANT: mandatory for trusted publishing @@ -94,7 +94,7 @@ jobs: - name: Read VERSION_NAME from dist/ id: version run: | - VERSION_NAME=$(basename dist/ably-*.tar.gz | sed -E 's/^ably-([^-]+)\.tar\.gz$/\1/') + VERSION_NAME=$(basename dist/ably_pubsub_server-*.tar.gz | sed -E 's/^ably_pubsub_server-([^-]+)\.tar\.gz$/\1/') echo "version=$VERSION_NAME" >> $GITHUB_OUTPUT - name: Compare version with tag @@ -118,7 +118,7 @@ jobs: environment: name: testpypi - url: https://test.pypi.org/p/ably + url: https://test.pypi.org/p/ably-pubsub-server permissions: id-token: write # IMPORTANT: mandatory for trusted publishing diff --git a/.gitignore b/.gitignore index 75ec0f34..426ac840 100644 --- a/.gitignore +++ b/.gitignore @@ -55,5 +55,5 @@ ably/types/options.py.orig test/ably/restsetup.py.orig .idea/**/* -ably/sync/** +ably/pubsub/sync/** test/ably/sync/** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ed650adb..b75ffbaa 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,7 +40,7 @@ The release process must include the following steps: 1. Ensure that all work intended for this release has landed to `main` 2. Create a release branch named like `release/2.0.1` -3. Add a commit to bump the version number, updating [`pyproject.toml`](./pyproject.toml) and [`ably/__init__.py`](./ably/__init__.py) +3. Add a commit to bump the version number, updating [`pyproject.toml`](./pyproject.toml) and [`ably/pubsub/version.py`](./ably/pubsub/version.py) 4. Run [`github_changelog_generator`](https://github.com/github-changelog-generator/github-changelog-generator) to automate the update of the [CHANGELOG](./CHANGELOG.md). This may require some manual intervention, both in terms of how the command is run and how the change log file is modified. Your mileage may vary: - The command you will need to run will look something like this: `github_changelog_generator -u ably -p ably-pubsub-python --since-tag v2.0.0 --output delta.md --token $GITHUB_TOKEN_WITH_REPO_ACCESS`. Generate token [here](https://github.com/settings/tokens/new?description=GitHub%20Changelog%20Generator%20token). - Using the command above, `--output delta.md` writes changes made after `--since-tag` to a new file diff --git a/LONG_DESCRIPTION.rst b/LONG_DESCRIPTION.rst index 518f2f86..54f7289c 100644 --- a/LONG_DESCRIPTION.rst +++ b/LONG_DESCRIPTION.rst @@ -1,7 +1,8 @@ -Official Ably Bindings for Python -================================== +Official Ably Pub/Sub Bindings for Python Servers +================================================= -A Python client library for Ably Realtime messaging. +A Python client library for Ably Pub/Sub realtime messaging, for use from +servers. Setup @@ -9,7 +10,13 @@ Setup You can install this package by using the pip tool and installing: - pip install ably + pip install ably-pubsub-server + +The package installs into ``ably.pubsub``. Both ``ably`` and ``ably.pubsub`` +are namespace packages shared with the other ``ably-*`` distributions, so the +whole public API is reached through ``ably.pubsub.server``:: + + from ably.pubsub.server import create_http_client, create_realtime_client Using Ably for Python diff --git a/README.md b/README.md index 5bfb89e0..7b81426f 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ ![Ably Pub/Sub Python Header](images/pythonSDK-github.png) -[![PyPI version](https://badge.fury.io/py/ably.svg)](https://pypi.org/project/ably/) +[![PyPI version](https://badge.fury.io/py/ably-pubsub-server.svg)](https://pypi.org/project/ably-pubsub-server/) [![License](https://img.shields.io/github/license/ably/ably-pubsub-python)](https://github.com/ably/ably-pubsub-python/blob/main/LICENSE) @@ -48,9 +48,31 @@ The following platforms are supported: To get started with your project, install the package: ```sh -pip install ably +pip install ably-pubsub-server ``` +The package installs into `ably.pubsub`, and the whole public API is reached +through `ably.pubsub.server`: + +```python +from ably.pubsub.server import create_http_client, create_realtime_client +``` + +Clients are built by these factories rather than by constructing a class, so +that the package a client comes from names the side your application runs on. +Annotate against the prototypes they return, `PubSubHttpClient` and +`PubSubRealtimeClient`. + +The synchronous, HTTP-only flavour lives alongside it: + +```python +from ably.pubsub.server.sync import create_http_client +``` + +Both `ably` and `ably.pubsub` are [namespace packages](https://peps.python.org/pep-0420/) +shared with the other `ably-*` distributions, so neither exports anything +itself — always import from `ably.pubsub.server`. + > [!NOTE] Install [Python](https://www.python.org/downloads/) version 3.8 or greater. @@ -59,8 +81,10 @@ Install [Python](https://www.python.org/downloads/) version 3.8 or greater. The following code connects to Ably's realtime messaging service, subscribes to a channel to receive messages, and publishes a test message to that same channel. ```python +from ably.pubsub.server import create_realtime_client + # Initialize Ably Realtime client -async with AblyRealtime('your-ably-api-key', client_id='me') as realtime_client: +async with create_realtime_client(key='your-ably-api-key', client_id='me') as realtime_client: # Wait for connection to be established await realtime_client.connection.once_async('connected') print('Connected to Ably') @@ -78,6 +102,53 @@ async with AblyRealtime('your-ably-api-key', client_id='me') as realtime_client: await channel.publish('test-event', 'hello world') ``` +### LiveObjects + +LiveObjects keeps shared, mutable state on a channel: maps and counters that every client +attached to it reads, updates and subscribes to. The channel needs the object modes, and +`channel.object.get()` attaches it, waits for the objects to sync, and returns the root map. +Values are read through a typed view of their path — `as_live_map()`, `as_live_counter()` or +`as_primitive()` — and reads are synchronous; writes are awaited. + +```python +from ably.pubsub.server import ChannelMode, ChannelOptions, create_realtime_client +from ably.pubsub.server.liveobjects import LiveCounter, LiveMap + +async with create_realtime_client(key='your-ably-api-key') as realtime_client: + channel = realtime_client.channels.get( + 'my-objects', + ChannelOptions(modes=[ChannelMode.OBJECT_SUBSCRIBE, ChannelMode.OBJECT_PUBLISH]), + ) + + # Attach, wait for the objects to sync, and get the root map + root = await channel.object.get() + + # Create objects by setting them on the root + await root.set('visits', LiveCounter.create(0)) + await root.set('profile', LiveMap.create({'name': 'Alice', 'theme': 'dark'})) + + # Read through a typed view of the path + visits = root.get('visits').as_live_counter() + print(visits.value()) # 0.0 + print(root.at('profile.name').as_primitive().value(str)) # Alice + + # Subscribe to changes at a path + def on_change(event): + print(f'{event.object.path()} changed') + + subscription = root.get('visits').subscribe(on_change) + + # Mutate + await visits.increment(5) + + # Batch several writes into a single message + async with root.get('profile').as_live_map().batch() as profile: + profile.set('name', 'Bob') + profile.remove('theme') + + subscription.unsubscribe() +``` + ## Releases The [CHANGELOG.md](https://github.com/ably/ably-pubsub-python/blob/main/CHANGELOG.md) contains details of the latest releases for this SDK. You can also view all Ably releases on [changelog.ably.com](https://changelog.ably.com). diff --git a/UPDATING.md b/UPDATING.md index 4b4dd719..64f69d56 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -1,5 +1,129 @@ # Upgrade / Migration Guide +## Version 3.x to 4.0.0 + +The package is now published to PyPI as `ably-pubsub-server` rather than `ably`, and +it installs into `ably.pubsub` rather than `ably`. Both `ably` and `ably.pubsub` are +[namespace packages](https://peps.python.org/pep-0420/) shared with the other +`ably-*` distributions, so neither exports anything of its own. The whole public +API is reached through `ably.pubsub.server`. + +### Installation + +Example 3.x: +```sh +pip install ably +``` + +Example 4.0.0: +```sh +pip install ably-pubsub-server +``` + +### Imports + +Example 3.x code: +```python +from ably import AblyRealtime, AblyRest +``` + +Example 4.0.0 code: +```python +from ably.pubsub.server import create_http_client, create_realtime_client +``` + +The synchronous, HTTP-only flavour moves the same way: + +Example 3.x code: +```python +from ably.sync import AblyRestSync +``` + +Example 4.0.0 code: +```python +from ably.pubsub.server.sync import create_http_client +``` + +Where 3.x left you reaching into submodules such as `ably.types.message` or +`ably.http.paginatedresult`, import from `ably.pubsub.server` instead. It is +the whole public API, and `ably.pubsub.server.sync` is its synchronous +counterpart. The packages beneath them are internal and free to move. + +For example, the encryption helpers move from `ably.util.crypto`: + +Example 3.x code: +```python +from ably.util.crypto import generate_random_key, get_default_params +``` + +Example 4.0.0 code: +```python +from ably.pubsub.server import generate_random_key, get_default_params +``` + +### Clients are built by factories + +`AblyRest` and `AblyRealtime` are internal in 4.0.0 and raise `TypeError` if +constructed directly. Build clients through the factories instead, which take the same options +as keyword arguments. The API key is no longer accepted positionally, so pass it as `key=`: + +Example 3.x code: +```python +rest = AblyRest(key='xxx') +realtime = AblyRealtime(key='xxx') +``` + +Example 4.0.0 code: +```python +pubsub_http_client = create_http_client(key='xxx') +pubsub_realtime_client = create_realtime_client(key='xxx') +``` + +### Type annotations + +The client classes are internal, so annotate against the prototypes the +factories are declared to return. Each is exported from the same module as the +factory that returns it. + +Example 3.x code: +```python +from ably import AblyRealtime, AblyRest +from ably.sync import AblyRestSync + +async def publish(client: AblyRest, name: str) -> None: ... +async def subscribe(client: AblyRealtime, name: str) -> None: ... +def publish_blocking(client: AblyRestSync, name: str) -> None: ... +``` + +Example 4.0.0 code: +```python +from ably.pubsub.server import PubSubHttpClient, PubSubRealtimeClient +from ably.pubsub.server.sync import PubSubHttpClient as PubSubHttpClientSync + +async def publish(client: PubSubHttpClient, name: str) -> None: ... +async def subscribe(client: PubSubRealtimeClient, name: str) -> None: ... +def publish_blocking(client: PubSubHttpClientSync, name: str) -> None: ... +``` + +### Releasing a realtime channel requires it to be detached + +`channels.release(name)` on a realtime client now raises an `AblyException` with code 90011 +if the channel is in any state other than `INITIALIZED`, `DETACHED` or `FAILED`, and leaves the +channel as it was. 3.x removed the channel from the collection whatever its state, which could +leave it attached in the Ably service, and later 3.x releases log a deprecation warning when +they do so. Detach the channel, and wait for that to complete, before releasing it: + +Example 3.x code: +```python +realtime.channels.release('my-channel') +``` + +Example 4.0.0 code: +```python +await pubsub_realtime_client.channels.get('my-channel').detach() +pubsub_realtime_client.channels.release('my-channel') +``` + ## Version 2.x to 3.0.0 The 3.0.0 version of ably-python introduces several breaking changes to improve the realtime experience and align the API with the Ably specification. These include: diff --git a/ably/__init__.py b/ably/__init__.py deleted file mode 100644 index 95412484..00000000 --- a/ably/__init__.py +++ /dev/null @@ -1,24 +0,0 @@ -import logging - -from ably.realtime.realtime import AblyRealtime -from ably.rest.auth import Auth -from ably.rest.push import Push -from ably.rest.rest import AblyRest -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.capability import Capability -from ably.types.channelmode import ChannelMode -from ably.types.channeloptions import ChannelOptions -from ably.types.channelsubscription import PushChannelSubscription -from ably.types.device import DeviceDetails -from ably.types.message import MessageAction, MessageVersion -from ably.types.operations import MessageOperation, PublishResult, UpdateDeleteResult -from ably.types.options import Options, VCDiffDecoder -from ably.util.crypto import CipherParams -from ably.util.exceptions import AblyAuthException, AblyException, IncompatibleClientIdException -from ably.vcdiff.defaultvcdiffdecoder import AblyVCDiffDecoder - -logger = logging.getLogger(__name__) -logger.addHandler(logging.NullHandler()) - -api_version = '5' -lib_version = '3.1.4' diff --git a/ably/http/__init__.py b/ably/pubsub/http/__init__.py similarity index 100% rename from ably/http/__init__.py rename to ably/pubsub/http/__init__.py diff --git a/ably/rest/annotations.py b/ably/pubsub/http/annotations.py similarity index 95% rename from ably/rest/annotations.py rename to ably/pubsub/http/annotations.py index 39c1664d..88332352 100644 --- a/ably/rest/annotations.py +++ b/ably/pubsub/http/annotations.py @@ -8,15 +8,15 @@ import msgpack -from ably.http.paginatedresult import PaginatedResult, format_params -from ably.types.annotation import ( +from ably.pubsub.request.paginatedresult import PaginatedResult, format_params +from ably.pubsub.types.annotation import ( Annotation, AnnotationAction, make_annotation_response_handler, ) -from ably.types.message import Message -from ably.types.options import Options -from ably.util.exceptions import AblyException +from ably.pubsub.types.message import Message +from ably.pubsub.types.options import Options +from ably.pubsub.util.exceptions import AblyException log = logging.getLogger(__name__) @@ -89,7 +89,7 @@ def construct_validate_annotation(msg_or_serial, annotation: Annotation) -> Anno ) -class RestAnnotations: +class HttpAnnotations: """ Provides REST API methods for managing annotations on messages. """ @@ -98,7 +98,7 @@ class RestAnnotations: def __init__(self, channel): """ - Initialize RestAnnotations. + Initialize HttpAnnotations. Args: channel: The REST Channel this annotations instance belongs to diff --git a/ably/rest/auth.py b/ably/pubsub/http/auth.py similarity index 96% rename from ably/rest/auth.py rename to ably/pubsub/http/auth.py index fe2aeae3..aa71d517 100644 --- a/ably/rest/auth.py +++ b/ably/pubsub/http/auth.py @@ -6,18 +6,18 @@ from datetime import timedelta from typing import TYPE_CHECKING -from ably.types.options import Options +from ably.pubsub.types.options import Options if TYPE_CHECKING: - from ably.realtime.realtime import AblyRealtime - from ably.rest.rest import AblyRest + from ably.pubsub.http.http import DefaultPubSubHttpClient + from ably.pubsub.realtime.realtime import DefaultPubSubRealtimeClient -from ably.types.capability import Capability -from ably.types.tokendetails import TokenDetails -from ably.types.tokenrequest import TokenRequest -from ably.util.clock import select_clock -from ably.util.exceptions import AblyAuthException, AblyException, IncompatibleClientIdException -from ably.util.helper import extract_url_params +from ably.pubsub.types.capability import Capability +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.types.tokenrequest import TokenRequest +from ably.pubsub.util.clock import select_clock +from ably.pubsub.util.exceptions import AblyAuthException, AblyException, IncompatibleClientIdException +from ably.pubsub.util.helper import extract_url_params __all__ = ["Auth"] @@ -29,7 +29,7 @@ class Method: BASIC = "BASIC" TOKEN = "TOKEN" - def __init__(self, ably: AblyRest | AblyRealtime, options: Options): + def __init__(self, ably: DefaultPubSubHttpClient | DefaultPubSubRealtimeClient, options: Options): self.__ably = ably self.__auth_options = options self.__clock = select_clock(options) @@ -409,7 +409,7 @@ async def token_request_from_auth_url(self, method: str, url: str, token_params, # Use clean URL for the request url = clean_url - from ably.http.http import Response + from ably.pubsub.request.http import Response resp = await self.ably.http.request_external( method=method, url=url, headers=headers, params=params, body=body) response = Response(resp) diff --git a/ably/rest/channel.py b/ably/pubsub/http/channel.py similarity index 94% rename from ably/rest/channel.py rename to ably/pubsub/http/channel.py index 7603a224..321ea275 100644 --- a/ably/rest/channel.py +++ b/ably/pubsub/http/channel.py @@ -1,27 +1,26 @@ -import base64 import json import logging -import os from collections import OrderedDict from typing import Iterator, Optional from urllib import parse import msgpack -from ably.http.paginatedresult import PaginatedResult, format_params -from ably.rest.annotations import RestAnnotations -from ably.types.channeldetails import ChannelDetails -from ably.types.message import ( +from ably.pubsub.http.annotations import HttpAnnotations +from ably.pubsub.request.paginatedresult import PaginatedResult, format_params +from ably.pubsub.types.channeldetails import ChannelDetails +from ably.pubsub.types.message import ( Message, MessageAction, MessageVersion, + assign_idempotent_ids, make_message_response_handler, make_single_message_response_handler, ) -from ably.types.operations import MessageOperation, PublishResult, UpdateDeleteResult -from ably.types.presence import Presence -from ably.util.crypto import get_cipher -from ably.util.exceptions import ( +from ably.pubsub.types.operations import MessageOperation, PublishResult, UpdateDeleteResult +from ably.pubsub.types.presence import Presence +from ably.pubsub.util.crypto import get_cipher +from ably.pubsub.util.exceptions import ( AblyException, IncompatibleClientIdException, catch_all, @@ -31,7 +30,7 @@ class Channel: - __annotations: RestAnnotations + __annotations: HttpAnnotations def __init__(self, ably, name, options): self.__ably = ably @@ -40,7 +39,7 @@ def __init__(self, ably, name, options): self.__cipher = None self.options = options self.__presence = Presence(self) - self.__annotations = RestAnnotations(self) + self.__annotations = HttpAnnotations(self) @catch_all async def history(self, direction=None, limit: int = None, start=None, end=None): @@ -58,11 +57,7 @@ def __publish_request_body(self, messages): """ # Idempotent publishing if self.ably.options.idempotent_rest_publishing: - # RSL1k1 - if all(message.id is None for message in messages): - base_id = base64.urlsafe_b64encode(os.urandom(12)).decode() - for serial, message in enumerate(messages): - message.id = f'{base_id}:{serial}' + assign_idempotent_ids(messages) request_body_list = [] for m in messages: @@ -370,7 +365,7 @@ def presence(self): return self.__presence @property - def annotations(self) -> RestAnnotations: + def annotations(self) -> HttpAnnotations: return self.__annotations @options.setter diff --git a/ably/rest/rest.py b/ably/pubsub/http/http.py similarity index 56% rename from ably/rest/rest.py rename to ably/pubsub/http/http.py index bc84e638..7d912d0b 100644 --- a/ably/rest/rest.py +++ b/ably/pubsub/http/http.py @@ -1,26 +1,42 @@ +import json import logging -from typing import Optional +from typing import List, Optional, Union from urllib.parse import urlencode -from ably.http.http import Http -from ably.http.paginatedresult import HttpPaginatedResponse, PaginatedResult, format_params -from ably.rest.auth import Auth -from ably.rest.channel import Channels -from ably.rest.push import Push -from ably.types.options import Options -from ably.types.stats import stats_response_processor -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException, catch_all +import msgpack + +from ably.pubsub.http.auth import Auth +from ably.pubsub.http.channel import Channels +from ably.pubsub.http.push import Push +from ably.pubsub.prototypes import PubSubHttpClient +from ably.pubsub.request.http import Http +from ably.pubsub.request.paginatedresult import HttpPaginatedResponse, PaginatedResult, format_params +from ably.pubsub.types.batch import ( + BatchPublishSpec, + BatchResult, + batch_presence_result_from_dict, + batch_publish_result_from_dict, +) +from ably.pubsub.types.message import assign_idempotent_ids +from ably.pubsub.types.options import Options +from ably.pubsub.types.stats import stats_response_processor +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.construction import reject_direct_construction +from ably.pubsub.util.exceptions import AblyException, catch_all log = logging.getLogger(__name__) -class AblyRest: - """Ably Rest Client""" +class DefaultPubSubHttpClient(PubSubHttpClient): + """The default :class:`~ably.pubsub.prototypes.PubSubHttpClient` implementation. + + Built by :func:`ably.pubsub.server.create_http_client`, which is the only + way to construct one. + """ def __init__(self, key: Optional[str] = None, token: Optional[str] = None, token_details: Optional[TokenDetails] = None, **kwargs): - """Create an AblyRest instance. + """Create an DefaultPubSubHttpClient instance. :Parameters: **Credentials** @@ -47,8 +63,9 @@ def __init__(self, key: Optional[str] = None, token: Optional[str] = None, - `auth_token`: Undocumented - `auth_callback`: Undocumented - `auth_url`: Undocumented - - `keep_alive`: use persistent connections. Defaults to True """ + reject_direct_construction(type(self), 'ably.pubsub.server.create_http_client') + if key is not None and ('key_name' in kwargs or 'key_secret' in kwargs): raise ValueError("key and key_name or key_secret are mutually exclusive. " "Provider either a key or key_name & key_secret") @@ -97,6 +114,63 @@ async def time(self, timeout: Optional[float] = None) -> float: AblyException.raise_for_response(r) return r.to_native()[0] + async def batch_publish(self, specs) -> Union[BatchResult, List[BatchResult]]: + """Publishes messages to several channels in a single request (RSC22) + + :Parameters: + - `specs`: a `BatchPublishSpec`, or a list of them. A dict with + `channels` and `messages` keys may stand in for a `BatchPublishSpec`. + + Returns a `BatchResult` holding a `BatchPublishSuccessResult` or a + `BatchPublishFailureResult` for each channel the spec names, or, given + a list of specs, a list of `BatchResult`s in the same order. + """ + single = not isinstance(specs, (list, tuple)) + specs = [BatchPublishSpec.factory(spec) for spec in ([specs] if single else specs)] + + for spec in specs: + if not spec.channels: + raise AblyException('A batch publish spec must name at least one channel', 400, 40000) + if not spec.messages: + raise AblyException('A batch publish spec must include at least one message', 400, 40000) + + # RSC22d: RSL1k1 applies to each spec separately + if self.options.idempotent_rest_publishing: + for spec in specs: + assign_idempotent_ids(spec.messages) + + binary = self.options.use_binary_protocol + body = [spec.as_dict(binary=binary) for spec in specs] + if single: + body = body[0] + if binary: + body = msgpack.packb(body, use_bin_type=True) + else: + body = json.dumps(body, separators=(',', ':')) + + response = await self.http.post('/messages', body=body) + + # RSC22b: the response is an array with a result for each spec, even when a single spec was sent + results = [BatchResult.from_dict(result, batch_publish_result_from_dict) + for result in response.to_native()] + return results[0] if single else results + + async def batch_presence(self, channels: List[str]) -> BatchResult: + """Retrieves the members present on several channels in a single request (RSC24) + + :Parameters: + - `channels`: the names of the channels + + Returns a `BatchResult` holding a `BatchPresenceSuccessResult` or a + `BatchPresenceFailureResult` for each channel. + """ + if isinstance(channels, str): + raise TypeError('Unexpected str channels, expected a list of channel names') + + path = '/presence?' + urlencode({'channels': ','.join(channels)}) + response = await self.http.get(path) + return BatchResult.from_dict(response.to_native(), batch_presence_result_from_dict) + @property def client_id(self) -> Optional[str]: return self.options.client_id diff --git a/ably/rest/push.py b/ably/pubsub/http/push.py similarity index 96% rename from ably/rest/push.py rename to ably/pubsub/http/push.py index f99b2b1d..afcea029 100644 --- a/ably/rest/push.py +++ b/ably/pubsub/http/push.py @@ -1,12 +1,12 @@ from typing import Optional -from ably.http.paginatedresult import PaginatedResult, format_params -from ably.types.channelsubscription import ( +from ably.pubsub.request.paginatedresult import PaginatedResult, format_params +from ably.pubsub.types.channelsubscription import ( PushChannelSubscription, channel_subscriptions_response_processor, channels_response_processor, ) -from ably.types.device import DeviceDetails, device_details_response_processor +from ably.pubsub.types.device import DeviceDetails, device_details_response_processor class Push: diff --git a/ably/pubsub/liveobjects/__init__.py b/ably/pubsub/liveobjects/__init__.py new file mode 100644 index 00000000..78b55146 --- /dev/null +++ b/ably/pubsub/liveobjects/__init__.py @@ -0,0 +1,7 @@ +"""LiveObjects: shared, conflict-free data structures stored on a realtime channel. + +The public names are re-exported from :mod:`ably.pubsub.server.liveobjects`; the +modules here hold the implementation behind them, and the internal classes the +specification names (``ObjectsPool``, ``InternalLiveMap``, ``InternalLiveCounter`` +and the wire types). +""" diff --git a/ably/pubsub/liveobjects/batch.py b/ably/pubsub/liveobjects/batch.py new file mode 100644 index 00000000..1d05afe0 --- /dev/null +++ b/ably/pubsub/liveobjects/batch.py @@ -0,0 +1,379 @@ +"""Batched writes, published as one message (RTPO20, RTINS17, RTBC*). + +`batch()` on a typed path or instance returns a `Batch`, an async context manager whose +`__aenter__` checks the write preconditions and resolves the target (RTPO20b-d, +RTINS17b-d) and returns the matching `BatchContext`. Inside the block every context +method is synchronous: reads resolve against the objects as they are held locally, so +they do not see the writes queued before them, and writes only queue. `__aexit__` +publishes everything queued as one message (RTBC16d) when the block exits without +raising, and closes the batch either way (RTPO20g). Any use of a context after its batch +has closed raises `AblyException` 40000 (RTBC16e). + +A write validates its arguments when it is called, so an invalid one raises inside the +block and nothing is published. A `LiveMap` or `LiveCounter` value is validated then too, +but evaluated only when the batch is published, because its object ids need the server +time (RTO16). +""" + +from __future__ import annotations + +from functools import partial +from typing import TYPE_CHECKING, Any, Awaitable, Callable, Generic, TypeVar, overload + +from ably.pubsub.liveobjects.enums import ValueType +from ably.pubsub.liveobjects.instance import ( + Instance, + LiveCounterInstance, + LiveMapInstance, + PrimitiveInstance, + expected_value_type, + primitive_value, + value_type_of, +) +from ably.pubsub.liveobjects.livecounter import counter_inc_message +from ably.pubsub.liveobjects.livemap import map_remove_message, map_set_message, map_set_messages +from ably.pubsub.liveobjects.objectmessage import ObjectMessage +from ably.pubsub.liveobjects.valuetypes import ( + LiveCounter, + LiveMap, + primitive_to_object_data, + validate_amount, + validate_key, + validate_value, +) +from ably.pubsub.util.exceptions import AblyException + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.realtimeobject import RealtimeObject + from ably.pubsub.liveobjects.valuetypes import JsonValue, LiveMapValue, Primitive, T + + # RTBC16b: builds the ObjectMessages of one queued write when the batch is flushed + MessageConstructor = Callable[[], Awaitable[list[ObjectMessage]]] + +C = TypeVar('C', bound='BatchContext') +V = TypeVar('V', bound='BatchContext') + + +class Batch(Generic[C]): + """RTBC16: the `async with` block a `batch()` call returns. + + `resolve` returns the value the batch acts on, or None, and is called when the block + is entered; the value must be the live object a `context_type` wraps. `description` + names the target in the error raised when it is not. A `Batch` opens one batch, so it + can be entered once. + """ + + def __init__(self, realtime_object: RealtimeObject, resolve: Callable[[], Any], context_type: type[C], + description: str): + self._realtime_object = realtime_object + self._resolve = resolve + self._context_type = context_type + self._description = description + self._root_context: RootBatchContext | None = None + + async def __aenter__(self) -> C: + """RTPO20b-d, RTINS17b-d: checks the write preconditions, resolves the target and opens + the batch. + + Raises AblyException 92007 if the target is not a live object of the type the batch + was taken on (RTPO20c, RTINS17c), and RuntimeError if this batch was entered before. + """ + if self._root_context is not None: + raise RuntimeError('This batch has already been entered; call batch() again for another') + self._realtime_object._check_write_preconditions() # RTPO20b, RTINS17b + + value = self._resolve() + expected = self._context_type._value_type + actual = value_type_of(value) if value is not None else None + if actual is not expected: + # RTPO20c, RTINS17c: nothing, a primitive or a live object of another type + reason = 'it does not resolve' if actual is None else f'it is of type {actual.value}' + raise AblyException(f'Cannot batch operations on {self._description} as a {expected.value}: ' + f'{reason}', 400, 92007) + + instance = Instance._wrap(self._realtime_object, value) + self._root_context = RootBatchContext(self._realtime_object, instance) # RTPO20d, RTINS17d + return self._root_context.wrap_instance(instance) + + async def __aexit__(self, exc_type, exc, tb) -> None: + """RTPO20f, RTPO20g, RTBC16d: publishes the queued writes unless the block raised, and + closes the batch either way. + + An exception raised in the block propagates, and nothing is published. + """ + if exc_type is not None: + self._root_context.close() # RTPO20g + return + await self._root_context.flush() # RTPO20f, RTINS17f + + +class RootBatchContext: + """RTBC16: the state one batch shares across every context opened from it.""" + + def __init__(self, realtime_object: RealtimeObject, instance: Instance): + self.realtime_object = realtime_object + self.instance = instance + self.wrapped_instances: dict[str, BatchContext] = {} # RTBC16a + self.queued_message_constructors: list[MessageConstructor] = [] # RTBC16b + self.closed = False + + def wrap_instance(self, instance: Instance) -> BatchContext: + """RTBC16c: the context for `instance`, one per object id. + + The context is the subclass matching the `Instance` subclass, so a `LiveMapInstance` + is wrapped in a `LiveMapBatchContext`. + """ + object_id = instance.id + if object_id is not None: + context = self.wrapped_instances.get(object_id) + if context is not None: + return context + + context = _context_type_for(instance)(self, instance) + if object_id is not None: + self.wrapped_instances[object_id] = context + return context + + def queue_message_constructor(self, constructor: MessageConstructor) -> None: + """RTBC16b: queues `constructor`, which builds the ObjectMessages of one write when the + batch is flushed.""" + self.queued_message_constructors.append(constructor) + + async def flush(self) -> None: + """RTBC16d: closes the batch, builds the queued messages in the order they were queued and + publishes them in one message through `RealtimeObject._publish_and_apply`, if there are + any. + + Raises whatever building or publishing the messages raises. Nothing is published if + building any of them fails. + """ + constructors = self.queued_message_constructors + self.close() + if constructors: + # The block may have awaited since its writes checked the write preconditions (RTO26) + self.realtime_object._check_write_preconditions() + object_messages: list[ObjectMessage] = [] + for construct in constructors: + object_messages.extend(await construct()) + if object_messages: + await self.realtime_object._publish_and_apply(object_messages) + + def close(self) -> None: + """RTBC16e: closes the batch, so that every later use of its contexts raises 40000, and + drops the contexts and writes it holds.""" + self.closed = True + self.wrapped_instances = {} + self.queued_message_constructors = [] + + +class BatchContext: + """RTBC1: a synchronous view of an `Instance` inside a batch (RTBC2a). + + As on an `Instance`, the view helpers are checked: asking for a type the context does + not wrap raises `AblyException` 92007 (RTBC1a, RTTS9d). Reads check the access + preconditions (RTO25), and writes the write preconditions (RTO26), before checking + that the batch is still open. + """ + + # The type of live object that a batch opening this subclass of context acts on + _value_type: ValueType | None = None + + def __init__(self, root_context: RootBatchContext, instance: Instance): + self._root_context = root_context # RTBC2b + self._instance = instance # RTBC2a + + @property + def id(self) -> str | None: + """RTBC3: the wrapped object's id, or None for a primitive.""" + self._throw_if_closed() # RTBC3b + return self._instance.id # RTBC3a + + def get(self, key: str) -> BatchContext | None: + """RTBC4: the context for the value at `key` of the wrapped map, or None. + + Raises AblyException 40003 if `key` is not a string. + """ + self._realtime_object._check_access_preconditions() # RTBC4b + self._throw_if_closed() # RTBC4c + instance = self._instance.get(key) # RTBC4d + if instance is None: + return None + return self._root_context.wrap_instance(instance) # RTBC4e + + def compact_json(self) -> JsonValue: + """RTBC11: `Instance.compact_json` on the wrapped instance.""" + self._realtime_object._check_access_preconditions() # RTBC11a + self._throw_if_closed() # RTBC11b + return self._instance.compact_json() # RTBC11c + + def as_live_map(self) -> LiveMapBatchContext: + """RTBC1a: this context as a map. Raises AblyException 92007 if it wraps something else.""" + self._throw_if_closed() # RTBC16e + return self._view(LiveMapBatchContext, self._instance.as_live_map()) # RTTS9d + + def as_live_counter(self) -> LiveCounterBatchContext: + """RTBC1a: this context as a counter. Raises AblyException 92007 if it wraps something else.""" + self._throw_if_closed() # RTBC16e + return self._view(LiveCounterBatchContext, self._instance.as_live_counter()) # RTTS9d + + def as_primitive(self) -> PrimitiveBatchContext: + """RTBC1a: this context as a primitive. Raises AblyException 92007 if it wraps something else.""" + self._throw_if_closed() # RTBC16e + return self._view(PrimitiveBatchContext, self._instance.as_primitive()) # RTTS9d + + @property + def _realtime_object(self) -> RealtimeObject: + return self._root_context.realtime_object + + def _view(self, context_type: type[V], instance: Instance) -> V: + """This context if it is a `context_type`, else a `context_type` wrapping `instance`.""" + if isinstance(self, context_type): + return self + return context_type(self._root_context, instance) + + def _throw_if_closed(self) -> None: + """RTBC16e: raises AblyException 40000 once the batch has closed.""" + if self._root_context.closed: + raise AblyException('Batch is closed: a batch context can only be used inside its batch block', + 400, 40000) + + def _queue(self, *object_messages: ObjectMessage) -> None: + """Queues a write whose ObjectMessages are already built.""" + async def construct() -> list[ObjectMessage]: + return list(object_messages) + + self._root_context.queue_message_constructor(construct) + + +class LiveMapBatchContext(BatchContext): + """RTBC1a: a batch context wrapping a map. + + A context of this type only ever wraps a map, so its writes never find another type + (RTBC12d, RTBC13d). + """ + + _instance: LiveMapInstance + _value_type = ValueType.LIVE_MAP + + def entries(self) -> list[tuple[str, BatchContext]]: + """RTBC6: `(key, context)` for each entry of the map, as `LiveMapInstance.entries` gives them.""" + self._realtime_object._check_access_preconditions() # RTBC6a + self._throw_if_closed() # RTBC6b + wrap_instance = self._root_context.wrap_instance + return [(key, wrap_instance(instance)) for key, instance in self._instance.entries()] # RTBC6c, RTBC6d + + def keys(self) -> list[str]: + """RTBC7: the keys of the map.""" + self._realtime_object._check_access_preconditions() # RTBC7a + self._throw_if_closed() # RTBC7b + return self._instance.keys() # RTBC7c + + def values(self) -> list[BatchContext]: + """RTBC8: a context for each entry of the map, as `entries` wraps them.""" + return [context for _, context in self.entries()] # RTBC8a-RTBC8c + + def size(self) -> int | None: + """RTBC9: the number of entries in the map. + + This is always an int, as the context always wraps a map; the signature is the one + `size()` has on the path views, where a path can resolve to nothing. + """ + self._realtime_object._check_access_preconditions() # RTBC9a + self._throw_if_closed() # RTBC9b + return self._instance.size() # RTBC9c + + def set(self, key: str, value: LiveMapValue) -> None: + """RTBC12: queues a MAP_SET of `key`, preceded by the creates a `LiveMap` or `LiveCounter` + value evaluates to. + + Raises AblyException 40003 for a key that is not a string and 40013 for a value of + an unsupported type (RTLM20e1), the contents of a `LiveMap` or `LiveCounter` value + included (RTLMV4). Such a value is evaluated when the batch is flushed, as its object + ids need the server time (RTO16); a primitive is encoded now, so later changes to a + dict or list the caller passed do not reach the batch. + """ + realtime_object = self._realtime_object + realtime_object._check_write_preconditions() # RTBC12b + self._throw_if_closed() # RTBC12c + validate_key(key) # RTLM20e1 + object_id = self._instance.id + + if not isinstance(value, (LiveCounter, LiveMap)): + self._queue(map_set_message(object_id, key, primitive_to_object_data(value))) # RTBC12e, RTLM20e1 + return + + validate_value(value) # RTLM20e1 + self._root_context.queue_message_constructor( + partial(map_set_messages, realtime_object, object_id, key, value)) # RTBC12e + + def remove(self, key: str) -> None: + """RTBC13: queues a MAP_REMOVE of `key`. + + Raises AblyException 40003 for a key that is not a string (RTLM21e1). + """ + self._realtime_object._check_write_preconditions() # RTBC13b + self._throw_if_closed() # RTBC13c + validate_key(key) # RTLM21e1 + self._queue(map_remove_message(self._instance.id, key)) # RTBC13e + + +class LiveCounterBatchContext(BatchContext): + """RTBC1a: a batch context wrapping a counter. + + A context of this type only ever wraps a counter, so its writes never find another + type (RTBC14d, RTBC15d). + """ + + _instance: LiveCounterInstance + _value_type = ValueType.LIVE_COUNTER + + def value(self) -> float: + """RTBC5: the counter's value.""" + self._realtime_object._check_access_preconditions() # RTBC5a + self._throw_if_closed() # RTBC5b + return self._instance.value() # RTBC5c + + def increment(self, amount: float = 1) -> None: + """RTBC14: queues a COUNTER_INC of `amount`. + + Raises AblyException 40003 unless `amount` is a finite number (RTLC12e1); a bool is + not a number. + """ + self._realtime_object._check_write_preconditions() # RTBC14b + self._throw_if_closed() # RTBC14c + self._queue(counter_inc_message(self._instance.id, validate_amount(amount))) # RTBC14e, RTLC12e1 + + def decrement(self, amount: float = 1) -> None: + """RTBC15: queues a COUNTER_INC of `-amount`, after the same validation as `increment`.""" + self._realtime_object._check_write_preconditions() # RTBC15b + self._throw_if_closed() # RTBC15c + self.increment(-validate_amount(amount)) # RTBC15e + + +class PrimitiveBatchContext(BatchContext): + """RTBC1a: a batch context wrapping a primitive.""" + + @overload + def value(self) -> Primitive: ... + + @overload + def value(self, expected: type[T]) -> T | None: ... + + def value(self, expected: type | None = None) -> Any: + """RTBC5: the wrapped primitive; with `expected`, only if it is of that type, as for + `PrimitiveInstance.value`.""" + value_type = expected_value_type(expected) + self._realtime_object._check_access_preconditions() # RTBC5a + self._throw_if_closed() # RTBC5b + return primitive_value(self._instance._value, value_type) # RTBC5c + + +def _context_type_for(instance: Instance) -> type[BatchContext]: + """RTBC1a: the `BatchContext` subclass matching the `Instance` subclass of `instance`.""" + if isinstance(instance, LiveMapInstance): + return LiveMapBatchContext + if isinstance(instance, LiveCounterInstance): + return LiveCounterBatchContext + if isinstance(instance, PrimitiveInstance): + return PrimitiveBatchContext + return BatchContext diff --git a/ably/pubsub/liveobjects/defaults.py b/ably/pubsub/liveobjects/defaults.py new file mode 100644 index 00000000..fbfd3c1e --- /dev/null +++ b/ably/pubsub/liveobjects/defaults.py @@ -0,0 +1,10 @@ +"""Default values for the LiveObjects client.""" + +# RTO10a: how often tombstoned objects and map entries are checked for release +GC_INTERVAL_MS = 5 * 60 * 1000 + +# RTO10b3: the grace period used when ConnectionDetails.objectsGCGracePeriod is absent +GC_GRACE_PERIOD_MS = 24 * 60 * 60 * 1000 + +# RTO3b: the object id of the root map, which every ObjectsPool holds +ROOT_OBJECT_ID = 'root' diff --git a/ably/pubsub/liveobjects/enums.py b/ably/pubsub/liveobjects/enums.py new file mode 100644 index 00000000..b42e90ee --- /dev/null +++ b/ably/pubsub/liveobjects/enums.py @@ -0,0 +1,39 @@ +"""The enumerations LiveObjects shares across its modules.""" + +from enum import Enum + + +class ObjectsSyncState(str, Enum): + """RTO17a: the state of synchronising a channel's objects with Ably.""" + + INITIALIZED = 'initialized' # RTO17a1 + SYNCING = 'syncing' # RTO17a2 + SYNCED = 'synced' # RTO17a3 + + +class ObjectsEvent(str, Enum): + """RTO18b: the sync state events `RealtimeObject.on` registers listeners for.""" + + SYNCING = 'syncing' # RTO18b1 + SYNCED = 'synced' # RTO18b2 + + +class ObjectsOperationSource(str, Enum): + """RTO22: where an operation being applied came from.""" + + LOCAL = 'local' # RTO22a: applied on receipt of the ACK for a local publish + CHANNEL = 'channel' # RTO22b: received over the channel + + +class ValueType(Enum): + """RTTS2: the category of value a path or an instance holds.""" + + STRING = 'string' # RTTS2a1 + NUMBER = 'number' # RTTS2a2 + BOOLEAN = 'boolean' # RTTS2a3 + BINARY = 'binary' # RTTS2a4 + JSON_OBJECT = 'json_object' # RTTS2a5 + JSON_ARRAY = 'json_array' # RTTS2a6 + LIVE_MAP = 'live_map' # RTTS2a7 + LIVE_COUNTER = 'live_counter' # RTTS2a8 + UNKNOWN = 'unknown' # RTTS2a9 diff --git a/ably/pubsub/liveobjects/instance.py b/ably/pubsub/liveobjects/instance.py new file mode 100644 index 00000000..3164ce4b --- /dev/null +++ b/ably/pubsub/liveobjects/instance.py @@ -0,0 +1,394 @@ +"""Identity-addressed views onto a channel's objects (RTINS*, RTTS7-RTTS10). + +An `Instance` wraps one resolved value, a live object or a primitive, and follows that +object wherever it sits in the graph. Every instance is one of `LiveMapInstance`, +`LiveCounterInstance` or `PrimitiveInstance`, matching the value it wraps. The view +helpers are checked: asking for a type the instance does not wrap raises `AblyException` +92007 (RTTS9d). + +The module also holds what the path and instance views share about resolved values: +their `ValueType`, the type filter `value(expected)` applies, and compaction. +""" + +from __future__ import annotations + +import base64 +import copy +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, Callable, overload + +from ably.pubsub.liveobjects.enums import ValueType +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.livemap import InternalLiveMap +from ably.pubsub.liveobjects.liveobject import LiveObject +from ably.pubsub.liveobjects.publicmessage import ObjectMessage +from ably.pubsub.util.exceptions import AblyException + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.batch import Batch, LiveCounterBatchContext, LiveMapBatchContext + from ably.pubsub.liveobjects.liveobject import LiveObjectUpdate + from ably.pubsub.liveobjects.realtimeobject import RealtimeObject + from ably.pubsub.liveobjects.subscription import Subscription + from ably.pubsub.liveobjects.valuetypes import JsonObject, JsonValue, LiveMapValue, Primitive, T + +# The types `value(expected)` accepts as `expected`, and the category each one selects +_EXPECTED_VALUE_TYPES: tuple[tuple[type, ValueType], ...] = ( + (str, ValueType.STRING), + (float, ValueType.NUMBER), + (bool, ValueType.BOOLEAN), + (bytes, ValueType.BINARY), + (list, ValueType.JSON_ARRAY), + (dict, ValueType.JSON_OBJECT), +) + +_PRIMITIVE_VALUE_TYPES = frozenset(value_type for _, value_type in _EXPECTED_VALUE_TYPES) + + +def value_type_of(value: Any) -> ValueType: + """RTTS2: the category of a resolved value, `UNKNOWN` for one in no known category (RTTS2a9). + + A boolean is never a number. + """ + if isinstance(value, InternalLiveMap): + return ValueType.LIVE_MAP + if isinstance(value, InternalLiveCounter): + return ValueType.LIVE_COUNTER + if isinstance(value, bool): + return ValueType.BOOLEAN + if isinstance(value, (int, float)): + return ValueType.NUMBER + if isinstance(value, str): + return ValueType.STRING + if isinstance(value, (bytes, bytearray)): + return ValueType.BINARY + if isinstance(value, dict): + return ValueType.JSON_OBJECT + if isinstance(value, list): + return ValueType.JSON_ARRAY + return ValueType.UNKNOWN + + +def expected_value_type(expected: type | None) -> ValueType | None: + """The category `value(expected)` filters on, or None for no filter. + + Raises TypeError for an `expected` other than `str`, `float`, `bool`, `bytes`, `list` + or `dict`. + """ + if expected is None: + return None + for python_type, value_type in _EXPECTED_VALUE_TYPES: + if expected is python_type: + return value_type + raise TypeError(f'expected must be one of str, float, bool, bytes, list or dict, not {expected!r}') + + +def primitive_value(value: Any, value_type: ValueType | None = None) -> Any: + """RTPO7d, RTTS6c: `value` if it is a primitive, of category `value_type` when one is given, + else None. + + A number is a float, and a JSON object or array is a copy, so that changing it leaves + the object it was read from unchanged. + """ + if value is None: + return None + actual_type = value_type_of(value) + if actual_type not in _PRIMITIVE_VALUE_TYPES or (value_type is not None and actual_type is not value_type): + return None + if actual_type is ValueType.NUMBER: + return float(value) + if actual_type is ValueType.BINARY: + return bytes(value) + if actual_type in (ValueType.JSON_OBJECT, ValueType.JSON_ARRAY): + return copy.deepcopy(value) + return value + + +def compact_json_value(value: Any) -> JsonValue: + """RTPO14, RTINS11: a JSON-serializable snapshot of a resolved value, a map as + `compact_json_map` builds it.""" + if isinstance(value, InternalLiveMap): + return compact_json_map(value) + return _compact_json_leaf(value) + + +def compact_json_map(live_map: InternalLiveMap) -> JsonObject: + """RTPO13c, RTPO14b: a JSON object of a map's entries that are not tombstoned (RTPO13c1). + + Nested maps become nested objects (RTPO13c2) and counters their values (RTPO13c3). Maps + are visited depth first in entry order, and a map reached again in the same snapshot, + through a cycle or a second reference, is `{'objectId': id}` (RTPO13c5, RTPO14b2). + """ + result: JsonObject = {} + visited = {live_map.object_id} + # A stack in place of recursion, so that the depth of the tree is not limited by the + # interpreter's. Each item is an object being built and the entries of its map still to add. + stack = [(result, iter(live_map.entries()))] + while stack: + target, entries = stack[-1] + for key, entry_value in entries: + if not isinstance(entry_value, InternalLiveMap): + target[key] = _compact_json_leaf(entry_value) + continue + if entry_value.object_id in visited: + target[key] = {'objectId': entry_value.object_id} + continue + child: JsonObject = {} + visited.add(entry_value.object_id) + target[key] = child + stack.append((child, iter(entry_value.entries()))) + break + else: + stack.pop() + return result + + +def _compact_json_leaf(value: Any) -> JsonValue: + """RTPO13c3, RTPO13c4, RTPO13d, RTPO13e, RTPO14b1: the snapshot of a value that is not a map. + + A counter is its value, binary a base64 string, and a JSON object or array a copy. + """ + if isinstance(value, InternalLiveCounter): + return value.value() + if isinstance(value, (bytes, bytearray)): + return base64.b64encode(value).decode('ascii') + if isinstance(value, (dict, list)): + return copy.deepcopy(value) + return value + + +@dataclass +class InstanceSubscriptionEvent: + """RTINS16e: what an `Instance.subscribe` listener receives.""" + + object: Instance # RTINS16e1 + message: ObjectMessage | None = None # RTINS16e2 + + +class Instance: + """RTINS1, RTTS7: a reference to a specific live object or primitive value.""" + + def __init__(self, realtime_object: RealtimeObject, value: LiveObject | Primitive): + self._realtime_object = realtime_object + self._value = value # RTINS2a + + def __repr__(self) -> str: + if isinstance(self._value, LiveObject): + return f'{type(self).__name__}(id={self._value.object_id!r})' + return f'{type(self).__name__}({self._value!r})' + + @staticmethod + def _wrap(realtime_object: RealtimeObject, value: LiveObject | Primitive) -> Instance: + """The `Instance` subclass matching `value`.""" + if isinstance(value, InternalLiveMap): + return LiveMapInstance(realtime_object, value) + if isinstance(value, InternalLiveCounter): + return LiveCounterInstance(realtime_object, value) + return PrimitiveInstance(realtime_object, value) + + @property + def id(self) -> str | None: + """RTINS3: the wrapped object's id, or None for a primitive.""" + if isinstance(self._value, LiveObject): + return self._value.object_id # RTINS3a + return None # RTINS3b + + @property + def type(self) -> ValueType: + """RTTS8a: the type of the wrapped value.""" + return value_type_of(self._value) + + def get(self, key: str) -> Instance | None: + """RTINS5: an instance wrapping the value at `key` of the wrapped map, or None. + + Raises AblyException 40003 if `key` is not a string. + """ + self._realtime_object._check_access_preconditions() # RTINS5b + if not isinstance(key, str): + raise AblyException(f'Map key must be a string, not {type(key).__name__}', 400, 40003) + if not isinstance(self._value, InternalLiveMap): + return None # RTINS5d + value = self._value.get(key) + if value is None: + return None + return Instance._wrap(self._realtime_object, value) # RTINS5c + + def compact_json(self) -> JsonValue: + """RTINS11: a JSON-serializable snapshot of the wrapped value, as + `PathObject.compact_json` builds it. Never None (RTINS11c).""" + self._realtime_object._check_access_preconditions() # RTINS11a + return compact_json_value(self._value) # RTINS11b + + def as_live_map(self) -> LiveMapInstance: + """RTTS9a: this instance as a map. Raises AblyException 92007 if it wraps something else.""" + if not isinstance(self._value, InternalLiveMap): + raise self._view_mismatch('a map') # RTTS9d + if isinstance(self, LiveMapInstance): + return self + return LiveMapInstance(self._realtime_object, self._value) + + def as_live_counter(self) -> LiveCounterInstance: + """RTTS9b: this instance as a counter. Raises AblyException 92007 if it wraps something else.""" + if not isinstance(self._value, InternalLiveCounter): + raise self._view_mismatch('a counter') # RTTS9d + if isinstance(self, LiveCounterInstance): + return self + return LiveCounterInstance(self._realtime_object, self._value) + + def as_primitive(self) -> PrimitiveInstance: + """RTTS9c: this instance as a primitive. Raises AblyException 92007 if it wraps something else.""" + if isinstance(self._value, LiveObject): + raise self._view_mismatch('a primitive') # RTTS9d + if isinstance(self, PrimitiveInstance): + return self + return PrimitiveInstance(self._realtime_object, self._value) + + def _view_mismatch(self, requested: str) -> AblyException: + return AblyException(f'Cannot view an instance of type {self.type.value} as {requested}', 400, 92007) + + +class LiveMapInstance(Instance): + """RTTS10a: an instance wrapping an `InternalLiveMap`.""" + + _value: InternalLiveMap + + @property + def id(self) -> str: + """RTINS3a: the wrapped map's id.""" + return self._value.object_id + + def batch(self) -> Batch[LiveMapBatchContext]: + """RTINS17: a block whose queued writes are published as one message when it exits. + + Entering checks the write preconditions (RTINS17b). + """ + # Imported here, as the batch module builds on the views + from ably.pubsub.liveobjects.batch import Batch, LiveMapBatchContext + + return Batch(self._realtime_object, lambda: self._value, LiveMapBatchContext, f'object {self.id!r}') + + def compact_json(self) -> JsonObject: + """RTINS11, RTTS7a1: a JSON-serializable snapshot of the wrapped map.""" + self._realtime_object._check_access_preconditions() # RTINS11a + return compact_json_map(self._value) # RTINS11b + + def entries(self) -> list[tuple[str, Instance]]: + """RTINS6: `(key, instance)` for each entry of the map. + + An entry referencing an object this client does not hold has no value to wrap, and + is left out. + """ + self._realtime_object._check_access_preconditions() # RTINS6a + # RTINS6b + return [(key, Instance._wrap(self._realtime_object, value)) + for key, value in self._value.entries() if value is not None] + + def keys(self) -> list[str]: + """RTINS7: the keys of the map.""" + self._realtime_object._check_access_preconditions() # RTINS7a + return self._value.keys() # RTINS7b + + def values(self) -> list[Instance]: + """RTINS8: an instance for each entry of the map, as `entries` wraps them.""" + return [instance for _, instance in self.entries()] # RTINS8a, RTINS8b + + def size(self) -> int: + """RTINS9: the number of entries in the map.""" + self._realtime_object._check_access_preconditions() # RTINS9a + return self._value.size() # RTINS9b + + async def set(self, key: str, value: LiveMapValue) -> None: + """RTINS12: sets `key` in the wrapped map.""" + self._realtime_object._check_write_preconditions() # RTINS12b + await self._value.set(key, value) # RTINS12c + + async def remove(self, key: str) -> None: + """RTINS13: removes `key` from the wrapped map.""" + self._realtime_object._check_write_preconditions() # RTINS13b + await self._value.remove(key) # RTINS13c + + def subscribe(self, listener: Callable[[InstanceSubscriptionEvent], None]) -> Subscription: + """RTINS16: calls `listener` for each update to the wrapped map, wherever it sits (RTINS16g).""" + self._realtime_object._check_access_preconditions() # RTINS16b + return _subscribe(self, listener) + + +class LiveCounterInstance(Instance): + """RTTS10b: an instance wrapping an `InternalLiveCounter`.""" + + _value: InternalLiveCounter + + @property + def id(self) -> str: + """RTINS3a: the wrapped counter's id.""" + return self._value.object_id + + def batch(self) -> Batch[LiveCounterBatchContext]: + """RTINS17: a block whose queued writes are published as one message when it exits. + + Entering checks the write preconditions (RTINS17b). + """ + # Imported here, as the batch module builds on the views + from ably.pubsub.liveobjects.batch import Batch, LiveCounterBatchContext + + return Batch(self._realtime_object, lambda: self._value, LiveCounterBatchContext, f'object {self.id!r}') + + def value(self) -> float: + """RTINS4b: the wrapped counter's value.""" + self._realtime_object._check_access_preconditions() # RTINS4a + return self._value.value() + + def compact_json(self) -> float: + """RTINS11, RTTS7a3: the wrapped counter's value.""" + self._realtime_object._check_access_preconditions() # RTINS11a + return self._value.value() # RTINS11b + + async def increment(self, amount: float = 1) -> None: + """RTINS14: increments the wrapped counter by `amount`.""" + self._realtime_object._check_write_preconditions() # RTINS14b + await self._value.increment(amount) # RTINS14c + + async def decrement(self, amount: float = 1) -> None: + """RTINS15: decrements the wrapped counter by `amount`.""" + self._realtime_object._check_write_preconditions() # RTINS15b + await self._value.decrement(amount) # RTINS15c + + def subscribe(self, listener: Callable[[InstanceSubscriptionEvent], None]) -> Subscription: + """RTINS16: calls `listener` for each update to the wrapped counter, wherever it sits (RTINS16g).""" + self._realtime_object._check_access_preconditions() # RTINS16b + return _subscribe(self, listener) + + +class PrimitiveInstance(Instance): + """RTTS10c, RTTS10d: an instance wrapping a primitive value. It has no `subscribe` (RTTS7b).""" + + @overload + def value(self) -> Primitive: ... + + @overload + def value(self, expected: type[T]) -> T | None: ... + + def value(self, expected: type | None = None) -> Any: + """RTINS4c: the wrapped primitive; with `expected`, only if it is of that type, as for + `PrimitivePathObject.value`.""" + value_type = expected_value_type(expected) + self._realtime_object._check_access_preconditions() # RTINS4a + return primitive_value(self._value, value_type) + + +def _subscribe(instance: LiveMapInstance | LiveCounterInstance, + listener: Callable[[InstanceSubscriptionEvent], None]) -> Subscription: + """RTINS16d: subscribes `listener` to the updates of the live object `instance` wraps. + + The live object calls its listeners synchronously, logging any that raises, and drops + them once it is tombstoned (RTLO4b4c3c). + """ + realtime_object = instance._realtime_object + + def on_update(update: LiveObjectUpdate) -> None: + object_message = update.object_message + message = None + if object_message is not None and object_message.operation is not None: + message = ObjectMessage._from_internal(object_message, realtime_object._channel_name) # RTINS16e2 + listener(InstanceSubscriptionEvent(instance, message)) # RTINS16e1 + + return instance._value.subscribe(on_update) diff --git a/ably/pubsub/liveobjects/livecounter.py b/ably/pubsub/liveobjects/livecounter.py new file mode 100644 index 00000000..407eb7a9 --- /dev/null +++ b/ably/pubsub/liveobjects/livecounter.py @@ -0,0 +1,173 @@ +"""The counter CRDT (RTLC*).""" + +from __future__ import annotations + +import logging +from typing import TYPE_CHECKING + +from ably.pubsub.liveobjects.enums import ObjectsOperationSource +from ably.pubsub.liveobjects.liveobject import CounterUpdate, LiveCounterUpdate, LiveObject +from ably.pubsub.liveobjects.objectmessage import CounterInc, ObjectMessage, ObjectOperation, ObjectOperationAction +from ably.pubsub.liveobjects.valuetypes import validate_amount +from ably.pubsub.util.clock import Clock +from ably.pubsub.util.exceptions import AblyException + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.objectspool import ObjectsPool + +log = logging.getLogger(__name__) + + +class InternalLiveCounter(LiveObject): + """RTLC1: a counter holding a 64-bit float (RTLC3). + + A new counter has `data` 0 (RTLC4). + """ + + _update_type = LiveCounterUpdate + + def __init__(self, object_id: str, *, pool: ObjectsPool | None = None, clock: Clock | None = None): + super().__init__(object_id, pool=pool, clock=clock) + self.data: float = 0.0 # RTLC3, RTLC4b + + def value(self) -> float: + """RTLC5: the current count.""" + return self.data # RTLC5c + + async def increment(self, amount: float) -> None: + """RTLC12: publishes a COUNTER_INC of `amount` through `RealtimeObject._publish_and_apply`. + + Raises AblyException 40003 if `amount` is not a finite number (RTLC12e1); a bool + is not a number. + """ + await self._publish_counter_inc(validate_amount(amount)) + + async def decrement(self, amount: float) -> None: + """RTLC13: `increment` by `-amount`, after the same validation (RTLC13c).""" + await self._publish_counter_inc(-validate_amount(amount)) # RTLC13b, RTLC13c + + async def _publish_counter_inc(self, number: float) -> None: + realtime_object = self.realtime_object + if realtime_object is None: + raise AblyException('Unable to increment a counter that is not on a channel', 400, 40000) + await realtime_object._publish_and_apply([counter_inc_message(self.object_id, number)]) # RTLC12g + + def apply_operation(self, object_message: ObjectMessage, source: ObjectsOperationSource) -> bool: + """RTLC7: applies `object_message.operation`, returning whether it was applied (RTLC7g). + + The update is emitted through `notify_updated` (RTLC7d1a, RTLC7d5a, RTLC7d4c). + """ + if not self.can_apply_operation(object_message): + # RTLC7b; an operation with invalid serials has already been logged by can_apply_operation + if object_message.serial and object_message.site_code: + log.debug(f'InternalLiveCounter.apply_operation(): skipping an operation whose serial is not ' + f'newer than the one recorded for its site; serial={object_message.serial}, ' + f'site_code={object_message.site_code}, object_id={self.object_id}') + return False + + if source == ObjectsOperationSource.CHANNEL: + # RTLC7c: the serial is recorded whether or not the operation then applies + self.site_timeserials[object_message.site_code] = object_message.serial + + if self.is_tombstone: + return False # RTLC7e + + operation = object_message.operation + if operation.action == ObjectOperationAction.COUNTER_CREATE: + update = self.apply_counter_create(operation, object_message) # RTLC7d1 + elif operation.action == ObjectOperationAction.COUNTER_INC: + update = self.apply_counter_inc(operation.counter_inc, object_message) # RTLC7d5 + elif operation.action == ObjectOperationAction.OBJECT_DELETE: + update = self.tombstone(object_message) # RTLC7d4, RTLO5 + else: + # RTLC7d3 + log.warning(f'InternalLiveCounter.apply_operation(): skipping an object operation message with ' + f'an unsupported action; action={operation.action!r}, object_id={self.object_id}') + return False + + self.notify_updated(update) # RTLC7d1a, RTLC7d5a, RTLC7d4c + return True # RTLC7d1b, RTLC7d5b, RTLC7d4b + + def replace_data(self, object_message: ObjectMessage) -> LiveCounterUpdate: + """RTLC6: replaces this counter's data with `object_message.object`, returning the diff.""" + object_state = object_message.object + self.site_timeserials = dict(object_state.site_timeserials) # RTLC6a + + if self.is_tombstone: + return LiveCounterUpdate(noop=True) # RTLC6e, RTLC6e1 + + if object_state.tombstone: + return self.tombstone(object_message) # RTLC6f, RTLC6f2 + + previous_data = self.data # RTLC6g + self.create_operation_is_merged = False # RTLC6b + count = object_state.counter.count if object_state.counter is not None else None + self.data = count if count is not None else 0.0 # RTLC6c + if object_state.create_op is not None: + self.merge_initial_value(object_state.create_op, object_message) # RTLC6d + + update = self.diff(previous_data, self.data) # RTLC6h + update.object_message = object_message + return update + + def apply_counter_create(self, operation: ObjectOperation, object_message: ObjectMessage) -> LiveCounterUpdate: + """RTLC8: applies a COUNTER_CREATE.""" + if self.create_operation_is_merged: + # RTLC8b + log.debug(f'InternalLiveCounter.apply_counter_create(): skipping a COUNTER_CREATE for a counter ' + f'whose create operation is already merged; object_id={self.object_id}') + return LiveCounterUpdate(noop=True) + return self.merge_initial_value(operation, object_message) # RTLC8c, RTLC8e + + def apply_counter_inc(self, counter_inc: CounterInc | None, + object_message: ObjectMessage) -> LiveCounterUpdate: + """RTLC9: applies a COUNTER_INC. + + A COUNTER_INC with no `number`, or with no payload at all, is a no-op (RTLC9h). + """ + number = counter_inc.number if counter_inc is not None else None + if number is None: + return LiveCounterUpdate(noop=True) # RTLC9h + + self.data += number # RTLC9f + return LiveCounterUpdate(update=CounterUpdate(amount=number), object_message=object_message) # RTLC9g + + def merge_initial_value(self, operation: ObjectOperation, object_message: ObjectMessage) -> LiveCounterUpdate: + """RTLC16: merges the initial value of a create operation into this counter. + + The `CounterCreate` is `operation.resolved_counter_create`. + """ + counter_create = operation.resolved_counter_create + count = counter_create.count if counter_create is not None else None + self.create_operation_is_merged = True # RTLC16b + if count is None: + return LiveCounterUpdate(noop=True) # RTLC16d + + self.data += count # RTLC16a + return LiveCounterUpdate(update=CounterUpdate(amount=count), object_message=object_message) # RTLC16c + + def clear_data(self) -> None: + """RTO27a1: resets `data` to 0, emitting nothing.""" + self.data = 0.0 # RTLC4b + + @staticmethod + def diff(previous_data: float, new_data: float, *, for_tombstone: bool = False) -> LiveCounterUpdate: + """RTLC14: the update between two counter values. + + A zero difference is a no-op (RTLC14c), unless the diff is computed for a tombstone + (`for_tombstone`, RTLO4e5). + """ + amount = new_data - previous_data # RTLC14b + if amount == 0 and not for_tombstone: + return LiveCounterUpdate(noop=True) # RTLC14c + return LiveCounterUpdate(update=CounterUpdate(amount=amount)) + + +def counter_inc_message(object_id: str, number: float) -> ObjectMessage: + """RTLC12e: the COUNTER_INC of the counter `object_id` by `number`, a validated amount + (`validate_amount`).""" + return ObjectMessage(operation=ObjectOperation( + action=ObjectOperationAction.COUNTER_INC, # RTLC12e2 + object_id=object_id, # RTLC12e3 + counter_inc=CounterInc(number=number), # RTLC12e5 + )) diff --git a/ably/pubsub/liveobjects/livemap.py b/ably/pubsub/liveobjects/livemap.py new file mode 100644 index 00000000..85aadd15 --- /dev/null +++ b/ably/pubsub/liveobjects/livemap.py @@ -0,0 +1,451 @@ +"""The last-write-wins map CRDT (RTLM*).""" + +from __future__ import annotations + +import dataclasses +import logging +from typing import TYPE_CHECKING, Any + +from ably.pubsub.liveobjects.enums import ObjectsOperationSource +from ably.pubsub.liveobjects.liveobject import MAP_KEY_REMOVED, MAP_KEY_UPDATED, LiveMapUpdate, LiveObject +from ably.pubsub.liveobjects.objectmessage import ( + MapRemove, + MapSet, + ObjectData, + ObjectMessage, + ObjectOperation, + ObjectOperationAction, + ObjectsMapEntry, + ObjectsMapSemantics, +) +from ably.pubsub.liveobjects.valuetypes import ( + LiveCounter, + LiveMap, + evaluate, + primitive_to_object_data, + validate_key, + validate_value, +) +from ably.pubsub.util.clock import Clock +from ably.pubsub.util.exceptions import AblyException + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.objectspool import ObjectsPool + from ably.pubsub.liveobjects.realtimeobject import RealtimeObject + from ably.pubsub.liveobjects.valuetypes import LiveMapValue + +log = logging.getLogger(__name__) + + +class InternalLiveMap(LiveObject): + """RTLM1: a map of `ObjectsMapEntry` values by key, with per-entry last-write-wins (RTLM3). + + A new map has empty `data` and no `clear_timeserial` (RTLM4). A value read from the + map is a primitive, an `InternalLiveMap` or `InternalLiveCounter` looked up in `pool`, + or None. + """ + + _update_type = LiveMapUpdate + + def __init__(self, object_id: str, semantics: ObjectsMapSemantics = ObjectsMapSemantics.LWW, *, + pool: ObjectsPool | None = None, clock: Clock | None = None): + super().__init__(object_id, pool=pool, clock=clock) + self.semantics: ObjectsMapSemantics = semantics # RTLM4b + self.data: dict[str, ObjectsMapEntry] = {} # RTLM3, RTLM4c + self.clear_timeserial: str | None = None # RTLM25, RTLM4d + + def get(self, key: str) -> Any: + """RTLM5: the value at `key`, or None if there is none, it is tombstoned, or this map is.""" + if self.is_tombstone: + return None # RTLM5e + entry = self.data.get(key) + if entry is None: + return None # RTLM5d1 + if self.is_entry_tombstoned(entry): + return None # RTLM5d2h + return self._resolve(entry.data) + + def size(self) -> int: + """RTLM10: the number of entries that are not tombstoned (RTLM14).""" + return sum(1 for entry in self.data.values() if not self.is_entry_tombstoned(entry)) # RTLM10d + + def entries(self) -> list[tuple[str, Any]]: + """RTLM11: `(key, value)` for each entry that is not tombstoned, values as `get` returns them.""" + # RTLM11d1, RTLM11d3; a value that resolves to None is still returned (RTLM11d3a) + return [(key, self._resolve(entry.data)) for key, entry in self.data.items() + if not self.is_entry_tombstoned(entry)] + + def keys(self) -> list[str]: + """RTLM12: the keys `entries` returns.""" + return [key for key, _ in self.entries()] + + def values(self) -> list[Any]: + """RTLM13: the values `entries` returns.""" + return [value for _, value in self.entries()] + + async def set(self, key: str, value: LiveMapValue) -> None: + """RTLM20: publishes a MAP_SET, preceded by the creates a `LiveMap` or `LiveCounter` value + evaluates to, through `RealtimeObject._publish_and_apply`. + + Raises AblyException 40003 for a key that is not a string and 40013 for a value of + an unsupported type (RTLM20e1, RTLMV4b, RTLMV4c), before anything is requested or + published. + """ + validate_key(key) # RTLM20e1 + realtime_object = self._publishing_realtime_object() + object_messages = await map_set_messages(realtime_object, self.object_id, key, value) # RTLM20e + await realtime_object._publish_and_apply(object_messages) # RTLM20h1, RTLM20h2 + + async def remove(self, key: str) -> None: + """RTLM21: publishes a MAP_REMOVE through `RealtimeObject._publish_and_apply`. + + Raises AblyException 40003 for a key that is not a string (RTLM21e1). + """ + validate_key(key) # RTLM21e1 + realtime_object = self._publishing_realtime_object() + await realtime_object._publish_and_apply([map_remove_message(self.object_id, key)]) # RTLM21g + + def _publishing_realtime_object(self) -> RealtimeObject: + realtime_object = self.realtime_object + if realtime_object is None: + raise AblyException('Unable to update a map that is not on a channel', 400, 40000) + return realtime_object + + def is_entry_tombstoned(self, entry: ObjectsMapEntry) -> bool: + """RTLM14: whether `entry` is tombstoned, or references a tombstoned object in `pool`.""" + if entry.tombstone: + return True # RTLM14a + referenced = self._referenced_object(entry.data) + return referenced is not None and referenced.is_tombstone # RTLM14c, RTLM14b + + def apply_operation(self, object_message: ObjectMessage, source: ObjectsOperationSource) -> bool: + """RTLM15: applies `object_message.operation`, returning whether it was applied (RTLM15g). + + The update is emitted through `notify_updated` (RTLM15d1a and its siblings). + """ + if not self.can_apply_operation(object_message): + # RTLM15b; an operation with invalid serials has already been logged by can_apply_operation + if object_message.serial and object_message.site_code: + log.debug(f'InternalLiveMap.apply_operation(): skipping an operation whose serial is not ' + f'newer than the one recorded for its site; serial={object_message.serial}, ' + f'site_code={object_message.site_code}, object_id={self.object_id}') + return False + + if source == ObjectsOperationSource.CHANNEL: + # RTLM15c: the serial is recorded whether or not the operation then applies + self.site_timeserials[object_message.site_code] = object_message.serial + + if self.is_tombstone: + return False # RTLM15e + + operation = object_message.operation + action = operation.action + if action == ObjectOperationAction.MAP_CREATE: + update = self.apply_map_create(operation, object_message) # RTLM15d1 + elif action == ObjectOperationAction.MAP_SET: + map_set = operation.map_set + if map_set is None or map_set.key is None or map_set.value is None: + self._log_missing_payload(action) + return False + update = self.apply_map_set(map_set, object_message.serial, object_message) # RTLM15d6 + elif action == ObjectOperationAction.MAP_REMOVE: + map_remove = operation.map_remove + if map_remove is None or map_remove.key is None: + self._log_missing_payload(action) + return False + update = self.apply_map_remove(map_remove, object_message.serial, object_message.serial_timestamp, + object_message) # RTLM15d7 + elif action == ObjectOperationAction.OBJECT_DELETE: + update = self.tombstone(object_message) # RTLM15d5, RTLO5 + elif action == ObjectOperationAction.MAP_CLEAR: + update = self.apply_map_clear(object_message.serial, object_message) # RTLM15d8 + else: + # RTLM15d4 + log.warning(f'InternalLiveMap.apply_operation(): skipping an object operation message with an ' + f'unsupported action; action={action!r}, object_id={self.object_id}') + return False + + self.notify_updated(update) # RTLM15d1a, RTLM15d6a, RTLM15d7a, RTLM15d5c, RTLM15d8a + return True # RTLM15d1b, RTLM15d6b, RTLM15d7b, RTLM15d5b, RTLM15d8b + + def replace_data(self, object_message: ObjectMessage) -> LiveMapUpdate: + """RTLM6: replaces this map's data with `object_message.object`, returning the diff.""" + object_state = object_message.object + self.site_timeserials = dict(object_state.site_timeserials) # RTLM6a + + if self.is_tombstone: + return LiveMapUpdate(noop=True) # RTLM6e, RTLM6e1 + + if object_state.tombstone: + return self.tombstone(object_message) # RTLM6f, RTLM6f2 + + previous_data = self.data # RTLM6g + self.create_operation_is_merged = False # RTLM6b + objects_map = object_state.map + self.clear_timeserial = objects_map.clear_timeserial if objects_map is not None else None # RTLM6i + entries = objects_map.entries if objects_map is not None and objects_map.entries is not None else {} + # RTLM6c. Each entry is copied, so that applying operations to this map never alters the + # ObjectState it was replaced from. + self.data = {} + for key, entry in entries.items(): + entry = dataclasses.replace(entry, tombstoned_at=None) + if entry.tombstone: + entry.tombstoned_at = self._tombstoned_at(entry.serial_timestamp) # RTLM6c1 + self.data[key] = entry + if object_state.create_op is not None: + self.merge_initial_value(object_state.create_op, object_message) # RTLM6d + + update = self.diff(previous_data, self.data) # RTLM6h + update.object_message = object_message + return update + + def apply_map_create(self, operation: ObjectOperation, object_message: ObjectMessage) -> LiveMapUpdate: + """RTLM16: applies a MAP_CREATE.""" + if self.create_operation_is_merged: + # RTLM16b + log.debug(f'InternalLiveMap.apply_map_create(): skipping a MAP_CREATE for a map whose create ' + f'operation is already merged; object_id={self.object_id}') + return LiveMapUpdate(noop=True) + return self.merge_initial_value(operation, object_message) # RTLM16d, RTLM16f + + def apply_map_set(self, map_set: MapSet, serial: str | None, object_message: ObjectMessage) -> LiveMapUpdate: + """RTLM7: applies a MAP_SET for `map_set.key`.""" + key = map_set.key + if self._is_at_or_before_clear(serial): + # RTLM7h + log.debug(f'InternalLiveMap.apply_map_set(): skipping a MAP_SET at or before the map\'s clear ' + f'serial; key={key!r}, serial={serial}, clear_timeserial={self.clear_timeserial}, ' + f'object_id={self.object_id}') + return LiveMapUpdate(noop=True) + + entry = self.data.get(key) + if entry is not None: + if not self.can_apply_map_operation(entry.timeserial, serial): + # RTLM7a1 + log.debug(f'InternalLiveMap.apply_map_set(): skipping a MAP_SET not newer than the entry; ' + f'key={key!r}, serial={serial}, entry serial={entry.timeserial}, ' + f'object_id={self.object_id}') + return LiveMapUpdate(noop=True) + self._release_reference(entry.data, key) # RTLM7a3 + entry.data = map_set.value # RTLM7a2e + entry.timeserial = serial # RTLM7a2b + entry.tombstone = False # RTLM7a2c + entry.tombstoned_at = None # RTLM7a2d + else: + # RTLM7b4, RTLM7b2, RTLM7b3 + self.data[key] = ObjectsMapEntry(data=map_set.value, timeserial=serial, tombstone=False, + tombstoned_at=None) + + value = map_set.value + if value is not None and value.object_id and self.pool is not None: + # RTLM7g + referenced = self.pool.create_zero_value_object_if_not_exists(value.object_id) # RTLM7g1 + if referenced is not None: + referenced.add_parent_reference(self, key) # RTLM7g2 + + return LiveMapUpdate(update={key: MAP_KEY_UPDATED}, object_message=object_message) # RTLM7f + + def apply_map_remove(self, map_remove: MapRemove, serial: str | None, serial_timestamp: int | None, + object_message: ObjectMessage) -> LiveMapUpdate: + """RTLM8: applies a MAP_REMOVE for `map_remove.key`.""" + key = map_remove.key + if self._is_at_or_before_clear(serial): + # RTLM8g + log.debug(f'InternalLiveMap.apply_map_remove(): skipping a MAP_REMOVE at or before the map\'s ' + f'clear serial; key={key!r}, serial={serial}, clear_timeserial={self.clear_timeserial}, ' + f'object_id={self.object_id}') + return LiveMapUpdate(noop=True) + + entry = self.data.get(key) + if entry is not None: + if not self.can_apply_map_operation(entry.timeserial, serial): + # RTLM8a1 + log.debug(f'InternalLiveMap.apply_map_remove(): skipping a MAP_REMOVE not newer than the ' + f'entry; key={key!r}, serial={serial}, entry serial={entry.timeserial}, ' + f'object_id={self.object_id}') + return LiveMapUpdate(noop=True) + self._release_reference(entry.data, key) # RTLM8a3 + entry.data = None # RTLM8a2a + entry.timeserial = serial # RTLM8a2b + entry.tombstone = True # RTLM8a2c + entry.tombstoned_at = self._tombstoned_at(serial_timestamp) # RTLM8a2d + else: + # RTLM8b1, RTLM8b2, RTLM8b3 + self.data[key] = ObjectsMapEntry(data=None, timeserial=serial, tombstone=True, + tombstoned_at=self._tombstoned_at(serial_timestamp)) + + return LiveMapUpdate(update={key: MAP_KEY_REMOVED}, object_message=object_message) # RTLM8e + + def apply_map_clear(self, serial: str | None, object_message: ObjectMessage) -> LiveMapUpdate: + """RTLM24: applies a MAP_CLEAR.""" + if not serial: + # A MAP_CLEAR with no serial cannot be ordered against the map's entries. `apply_operation` + # never gets here, as it rejects a message with no serial (RTLO4a3); a direct call can. + log.warning(f'InternalLiveMap.apply_map_clear(): skipping a MAP_CLEAR with no serial; ' + f'object_id={self.object_id}') + return LiveMapUpdate(noop=True) + + if self.clear_timeserial is not None and self.clear_timeserial > serial: + # RTLM24c + log.debug(f'InternalLiveMap.apply_map_clear(): skipping a MAP_CLEAR older than the map\'s clear ' + f'serial; serial={serial}, clear_timeserial={self.clear_timeserial}, ' + f'object_id={self.object_id}') + return LiveMapUpdate(noop=True) + + self.clear_timeserial = serial # RTLM24d + removed: dict[str, str] = {} + for key, entry in list(self.data.items()): + # RTLM24e1 + if entry.timeserial is None or serial > entry.timeserial: + self._release_reference(entry.data, key) # RTLM24e1c + del self.data[key] # RTLM24e1a + removed[key] = MAP_KEY_REMOVED # RTLM24e1b + + return LiveMapUpdate(update=removed, object_message=object_message) # RTLM24f + + def merge_initial_value(self, operation: ObjectOperation, object_message: ObjectMessage) -> LiveMapUpdate: + """RTLM23: merges the initial entries of a create operation into this map. + + The `MapCreate` is `operation.resolved_map_create`. + """ + map_create = operation.resolved_map_create + merged: dict[str, str] = {} + entries = map_create.entries if map_create is not None else {} + for key, entry in entries.items(): + # The serial an entry is applied with is the entry's own, not the message's + if entry.tombstone: + # RTLM23a2 + update = self.apply_map_remove(MapRemove(key=key), entry.timeserial, entry.serial_timestamp, + object_message) + else: + # RTLM23a1 + update = self.apply_map_set(MapSet(key=key, value=entry.data), entry.timeserial, object_message) + if not update.noop: + merged.update(update.update) # RTLM23c + + self.create_operation_is_merged = True # RTLM23b + return LiveMapUpdate(update=merged, object_message=object_message) # RTLM23c + + def gc_tombstoned_entries(self, grace_period_ms: int, now_ms: int) -> None: + """RTLM19: removes tombstoned entries whose `tombstoned_at` is `grace_period_ms` or more + before `now_ms`.""" + for key, entry in list(self.data.items()): + # RTLM19a1 + if (entry.tombstone and entry.tombstoned_at is not None + and now_ms - entry.tombstoned_at >= grace_period_ms): + del self.data[key] + + def clear_data(self) -> None: + """RTO27a1: resets `data` to empty and `clear_timeserial` to None, emitting nothing. + + Each object an entry references stops recording this map as a parent (RTLO4e9). + """ + for key, entry in self.data.items(): + self._release_reference(entry.data, key) + self.data = {} # RTLM4c + self.clear_timeserial = None # RTLM4d + + @staticmethod + def can_apply_map_operation(entry_serial: str | None, operation_serial: str | None) -> bool: + """RTLM9: whether an operation with `operation_serial` may replace an entry with `entry_serial`.""" + if not entry_serial and not operation_serial: + return False # RTLM9b + if not entry_serial: + return True # RTLM9d + if not operation_serial: + return False # RTLM9c + return operation_serial > entry_serial # RTLM9a, RTLM9e + + @staticmethod + def diff(previous_data: dict[str, ObjectsMapEntry], new_data: dict[str, ObjectsMapEntry], *, + for_tombstone: bool = False) -> LiveMapUpdate: + """RTLM22: the update between two versions of a map's data. + + Only entries whose own `tombstone` flag is false are compared. An empty diff is a + no-op (RTLM22c), unless it is computed for a tombstone (`for_tombstone`, RTLO4e5). + """ + previous_live = {key: entry for key, entry in previous_data.items() if not entry.tombstone} + new_live = {key: entry for key, entry in new_data.items() if not entry.tombstone} + update: dict[str, str] = {} + for key in previous_live: + if key not in new_live: + update[key] = MAP_KEY_REMOVED # RTLM22b1 + for key, entry in new_live.items(): + if key not in previous_live or previous_live[key].data != entry.data: + update[key] = MAP_KEY_UPDATED # RTLM22b2, RTLM22b3 + + if not update and not for_tombstone: + return LiveMapUpdate(noop=True) # RTLM22c + return LiveMapUpdate(update=update) + + def _resolve(self, data: ObjectData | None) -> Any: + """RTLM5d2: the value `data` holds: a primitive, the object it references, or None.""" + if data is None: + return None + # RTLM5d2b-RTLM5d2e, and a JSON value (OD2g) + primitive = data.value + if primitive is not None: + return primitive + if data.object_id is not None: + return self._referenced_object(data) # RTLM5d2f1, RTLM5d2f2 + return None # RTLM5d2g + + def _referenced_object(self, data: ObjectData | None) -> LiveObject | None: + """The object in `pool` that `data` references, if there is one.""" + if data is None or data.object_id is None or self.pool is None: + return None + return self.pool.get(data.object_id) + + def _release_reference(self, data: ObjectData | None, key: str) -> None: + """RTLM7a3, RTLM8a3, RTLM24e1c, RTLO4e9: removes this map at `key` from the parent references of + the object `data` references, if it is in `pool`.""" + referenced = self._referenced_object(data) + if referenced is not None: + referenced.remove_parent_reference(self, key) + + def _is_at_or_before_clear(self, serial: str | None) -> bool: + """RTLM7h, RTLM8g: whether an operation with `serial` is no later than the map's last MAP_CLEAR.""" + return self.clear_timeserial is not None and (not serial or self.clear_timeserial >= serial) + + def _log_missing_payload(self, action: ObjectOperationAction) -> None: + log.warning(f'InternalLiveMap.apply_operation(): skipping a {action.name} operation with no key or ' + f'value; object_id={self.object_id}') + + +def map_set_message(object_id: str, key: str, data: ObjectData) -> ObjectMessage: + """RTLM20e: the MAP_SET of `key` in the map `object_id` to `data`.""" + return ObjectMessage(operation=ObjectOperation( + action=ObjectOperationAction.MAP_SET, # RTLM20e2 + object_id=object_id, # RTLM20e3 + map_set=MapSet(key=key, value=data), # RTLM20e6, RTLM20e7 + )) + + +def map_remove_message(object_id: str, key: str) -> ObjectMessage: + """RTLM21e: the MAP_REMOVE of `key` from the map `object_id`.""" + return ObjectMessage(operation=ObjectOperation( + action=ObjectOperationAction.MAP_REMOVE, # RTLM21e2 + object_id=object_id, # RTLM21e3 + map_remove=MapRemove(key=key), # RTLM21e5 + )) + + +async def map_set_messages(realtime_object: RealtimeObject, object_id: str, key: str, + value: LiveMapValue) -> list[ObjectMessage]: + """RTLM20e: the ObjectMessages setting `key` of the map `object_id` to `value`. + + For a `LiveMap` or `LiveCounter`, they are the creates it evaluates to, with object ids + generated from the server time (RTLM20e7g1), followed by a MAP_SET referencing the last + of them (RTLM20e7g2); for a primitive, just the MAP_SET (RTLM20e7b-RTLM20e7f). + + Raises as `validate_value` does for an invalid value (RTLM20e1), a blueprint before the + server time is fetched. + """ + if not isinstance(value, (LiveCounter, LiveMap)): + return [map_set_message(object_id, key, primitive_to_object_data(value))] + + validate_value(value) + server_time_ms = await realtime_object._get_server_time_ms() + object_messages = evaluate(value, server_time_ms) + data = ObjectData(object_id=object_messages[-1].operation.object_id) + return [*object_messages, map_set_message(object_id, key, data)] diff --git a/ably/pubsub/liveobjects/liveobject.py b/ably/pubsub/liveobjects/liveobject.py new file mode 100644 index 00000000..00a95ada --- /dev/null +++ b/ably/pubsub/liveobjects/liveobject.py @@ -0,0 +1,250 @@ +"""The behaviour every live object shares (RTLO*), and the updates objects emit.""" + +from __future__ import annotations + +import logging +from abc import ABC, abstractmethod +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Callable + +from ably.pubsub.liveobjects.defaults import ROOT_OBJECT_ID +from ably.pubsub.liveobjects.objectmessage import ObjectMessage +from ably.pubsub.liveobjects.subscription import Registry, Subscription +from ably.pubsub.util.clock import Clock + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.enums import ObjectsOperationSource + from ably.pubsub.liveobjects.livemap import InternalLiveMap + from ably.pubsub.liveobjects.objectspool import ObjectsPool + from ably.pubsub.liveobjects.realtimeobject import RealtimeObject + +log = logging.getLogger(__name__) + +# RTLM18b: the change recorded against a map key in a LiveMapUpdate +MAP_KEY_UPDATED = 'updated' +MAP_KEY_REMOVED = 'removed' + + +@dataclass +class LiveObjectUpdate: + """RTLO4b4: a change to a live object's data, as the object emits it.""" + + update: Any = None # RTLO4b4a + noop: bool = False # RTLO4b4b + object_message: ObjectMessage | None = None # RTLO4b4d + tombstone: bool = False # RTLO4b4e + + +@dataclass +class CounterUpdate: + """RTLC11b: what a counter update changed.""" + + amount: float # RTLC11b1 + + +@dataclass +class LiveCounterUpdate(LiveObjectUpdate): + """RTLC11: an update emitted by an `InternalLiveCounter`. + + `update` is a `CounterUpdate`, or None on a no-op. + """ + + update: CounterUpdate | None = None + + +@dataclass +class LiveMapUpdate(LiveObjectUpdate): + """RTLM18: an update emitted by an `InternalLiveMap`. + + `update` maps each changed key to `'updated'` or `'removed'` (RTLM18b). + """ + + update: dict[str, str] = field(default_factory=dict) + + +class LiveObject(ABC): + """RTLO1: the state and behaviour `InternalLiveCounter` and `InternalLiveMap` share. + + A live object is constructible on its own, with no channel, so that its CRDT + behaviour can be driven directly. `pool` is the `ObjectsPool` it reads other objects + from (RTLM5d2f, RTLM14c, RTLO4f); an object added to a pool with `pool[object_id] = obj` + is adopted by it. `clock` is what RTLO6b reads the local time from; without one the + object uses its pool's clock, and with neither, the system clock. + """ + + # The update type this object emits, which a no-op it returns is also an instance of + _update_type: type[LiveObjectUpdate] = LiveObjectUpdate + + def __init__(self, object_id: str, *, pool: ObjectsPool | None = None, clock: Clock | None = None): + self.object_id: str = object_id # RTLO3a + self.site_timeserials: dict[str, str] = {} # RTLO3b + self.create_operation_is_merged: bool = False # RTLO3c + self.is_tombstone: bool = False # RTLO3d + self.tombstoned_at: int | None = None # RTLO3e + self.parent_references: dict[str, set[str]] = {} # RTLO3f + self.pool: ObjectsPool | None = pool + self.data: Any = None + self._clock: Clock | None = clock + self._listeners: Registry[Callable[[LiveObjectUpdate], None]] = Registry() + + @property + def clock(self) -> Clock: + """The clock RTLO6b reads the local time from.""" + if self._clock is not None: + return self._clock + if self.pool is not None: + return self.pool.clock + return Clock() + + @property + def realtime_object(self) -> RealtimeObject | None: + """The `RealtimeObject` this object's pool belongs to, if any.""" + return self.pool.realtime_object if self.pool is not None else None + + def subscribe(self, listener: Callable[[LiveObjectUpdate], None]) -> Subscription: + """RTLO4b: registers `listener` for the updates this object emits. + + Listeners are called synchronously from `notify_updated`, in registration order. + """ + return Subscription(self._listeners.register(listener)) + + def notify_updated(self, update: LiveObjectUpdate) -> None: + """RTLO4b4c: emits `update`. + + Every update an operation produces passes through here, a no-op included, and a + no-op stops here (RTLO4b4c1). Otherwise the `subscribe` listeners are called + (RTLO4b4c3a), path subscriptions are dispatched through the `RealtimeObject`, if + there is one (RTLO4b4c3b), and a tombstone update then deregisters the `subscribe` + listeners (RTLO4b4c3c). A listener registered during the dispatch is not called by it, + and one deregistered during it is not called again. + """ + if update.noop: + return + + for registration in self._listeners.snapshot(): + if not registration.active: + continue + try: + registration.item(update) + except Exception: + log.exception(f'LiveObject.notify_updated(): a subscription listener raised; ' + f'object_id={self.object_id}') + + realtime_object = self.realtime_object + if realtime_object is not None: + realtime_object._path_object_subscription_register.dispatch(self, update) + + if update.tombstone: + self._listeners.clear() + + def can_apply_operation(self, object_message: ObjectMessage) -> bool: + """RTLO4a: whether `object_message`'s serial is newer than this object's for its site.""" + serial = object_message.serial + site_code = object_message.site_code + # RTLO4a3 + if not isinstance(serial, str) or not serial or not isinstance(site_code, str) or not site_code: + log.warning(f'LiveObject.can_apply_operation(): object operation message has invalid serial ' + f'values, skipping it; serial={serial!r}, site_code={site_code!r}, ' + f'object_id={self.object_id}') + return False + + site_serial = self.site_timeserials.get(site_code) # RTLO4a4 + if not site_serial: + return True # RTLO4a5 + return serial > site_serial # RTLO4a6 + + @abstractmethod + def apply_operation(self, object_message: ObjectMessage, source: ObjectsOperationSource) -> bool: + """RTLC7, RTLM15: applies `object_message.operation`, returning whether it was applied. + + The resulting update is emitted through `notify_updated`, not returned. + """ + + @abstractmethod + def replace_data(self, object_message: ObjectMessage) -> LiveObjectUpdate: + """RTLC6, RTLM6: replaces this object's data with `object_message.object`. + + The update is returned, not emitted; the caller emits it (RTO5c7). + """ + + def tombstone(self, object_message: ObjectMessage) -> LiveObjectUpdate: + """RTLO4e: tombstones this object, returning the resulting update. + + The update has `tombstone` set and `object_message` populated, and is never a no-op, + except for the root object, which is never tombstoned (RTLO4e10). + """ + if self.object_id == ROOT_OBJECT_ID: + log.warning(f'LiveObject.tombstone(): attempt to tombstone the root object was rejected; ' + f'serial={object_message.serial}, site_code={object_message.site_code}, ' + f'message id={object_message.id}') + return self._update_type(noop=True) + + self.is_tombstone = True # RTLO4e2 + self.tombstoned_at = self._tombstoned_at(object_message.serial_timestamp) # RTLO4e3 + previous_data = self.data + self.clear_data() # RTLO4e9, RTLO4e4 + update = self.diff(previous_data, self.data, for_tombstone=True) # RTLO4e5 + update.tombstone = True # RTLO4e6 + update.object_message = object_message # RTLO4e7 + return update + + @abstractmethod + def clear_data(self) -> None: + """RTO27a1: resets this object's data to that of a new empty object, emitting nothing.""" + + @staticmethod + @abstractmethod + def diff(previous_data: Any, new_data: Any, *, for_tombstone: bool = False) -> LiveObjectUpdate: + """RTLC14, RTLM22: the update between two versions of this type of object's data.""" + + def add_parent_reference(self, parent: InternalLiveMap, key: str) -> None: + """RTLO4g: records that `parent` references this object at `key`.""" + self.parent_references.setdefault(parent.object_id, set()).add(key) + + def remove_parent_reference(self, parent: InternalLiveMap, key: str) -> None: + """RTLO4h: removes the record that `parent` references this object at `key`.""" + keys = self.parent_references.get(parent.object_id) + if keys is None: + return # RTLO4h1 + keys.discard(key) # RTLO4h2 + if not keys: + del self.parent_references[parent.object_id] # RTLO4h3 + + def get_full_paths(self) -> list[list[str]]: + """RTLO4f: every key-path from the root map to this object, each once, in no set order. + + Parents are looked up by object id in this object's pool. + """ + paths: list[list[str]] = [] + seen: set[tuple[str, ...]] = set() + # Each item is an object, the keys from it down to this object, and the ids of the + # objects already on that path, which a simple path does not revisit (RTLO4f2) + stack: list[tuple[LiveObject, list[str], frozenset[str]]] = [(self, [], frozenset())] + while stack: + live_object, path, visited = stack.pop() + if live_object.object_id in visited: + continue + if live_object.object_id == ROOT_OBJECT_ID: + if tuple(path) not in seen: + seen.add(tuple(path)) + paths.append(path) + continue + if self.pool is None: + continue + visited = visited | {live_object.object_id} + for parent_id, keys in live_object.parent_references.items(): + parent = self.pool.get(parent_id) + if parent is None: + continue + for key in keys: + stack.append((parent, [key, *path], visited)) + return paths + + def _tombstoned_at(self, serial_timestamp: int | None) -> int: + """RTLO6: the time something is tombstoned at, given the operation's `serial_timestamp`.""" + if serial_timestamp is not None: + return serial_timestamp # RTLO6a + # RTLO6b, RTLO6b1 + log.debug(f'LiveObject._tombstoned_at(): no serial_timestamp for the operation, using the local ' + f'clock instead; object_id={self.object_id}') + return self.clock.now_ms() diff --git a/ably/pubsub/liveobjects/objectid.py b/ably/pubsub/liveobjects/objectid.py new file mode 100644 index 00000000..8a7a349a --- /dev/null +++ b/ably/pubsub/liveobjects/objectid.py @@ -0,0 +1,31 @@ +"""Object ids generated by the client (RTO14) and parsed by it (RTO6b).""" + +from __future__ import annotations + +import base64 +import hashlib +import os + +# 12 random bytes base64-encode to 16 characters, the RTLCV4d/RTLMV4g minimum +_NONCE_ENTROPY_BYTES = 12 + + +def generate_object_id(object_type: str, initial_value: str, nonce: str, timestamp_ms: int) -> str: + """RTO14: `[type]:[hash]@[timestamp]`, where the hash is the URL-safe, unpadded base64 of + the SHA-256 of `initial_value:nonce` (RTO14b). + + `object_type` is `'map'` or `'counter'` (RTO14a1). + """ + digest = hashlib.sha256(f'{initial_value}:{nonce}'.encode()).digest() # RTO14b1 + hash_part = base64.urlsafe_b64encode(digest).decode('ascii').rstrip('=') # RTO14b2 + return f'{object_type}:{hash_part}@{int(timestamp_ms)}' # RTO14c + + +def generate_nonce() -> str: + """RTLCV4d, RTLMV4g: a random string of at least 16 characters.""" + return base64.urlsafe_b64encode(os.urandom(_NONCE_ENTROPY_BYTES)).decode('ascii') + + +def object_type_of(object_id: str) -> str: + """RTO6b1: the type an object id names, the part before the first `:`.""" + return object_id.split(':', 1)[0] diff --git a/ably/pubsub/liveobjects/objectmessage.py b/ably/pubsub/liveobjects/objectmessage.py new file mode 100644 index 00000000..cde5368a --- /dev/null +++ b/ably/pubsub/liveobjects/objectmessage.py @@ -0,0 +1,679 @@ +"""The internal object message types and their wire encoding. + +These are the `ObjectMessage` (OM*), `ObjectOperation` (OOP*), `ObjectState` (OST*) +and `ObjectData` (OD*) types of features.md, with the protocol v6 operation payloads +(`MapCreate`, `MapSet`, `MapRemove`, `CounterCreate`, `CounterInc`, `ObjectDelete`, +`MapClear`, `MapCreateWithObjectId`, `CounterCreateWithObjectId`) and the `ObjectsMap`, +`ObjectsMapEntry` and `ObjectsCounter` state types. + +Every type decodes from a wire dictionary with `from_dict(obj, format)` and encodes to +one with `to_dict(format)`, where `format` is `'json'` or `'msgpack'`. The two forms +differ only in how values are carried: + +- `ObjectData.bytes` is a base64 string on the JSON wire and raw binary on msgpack + (OD2d, OD4c2, OD4d2, OD5a1, OD5b2). +- `ObjectData.json` is a JSON-encoded string on both wires (OD2g), and a decoded `dict` + or `list` here. +- Numbers are float64 (OD4c3) and decode to `float`. +- Enums are carried as their integer wire values (OOP2, OMP2). + +Decoded values are what the rest of the package works with: `ObjectData.bytes` holds +`bytes`, `ObjectData.json` holds the parsed value, and wire names are snake_cased. +Fields that are only ever held locally (`ObjectsMapEntry.tombstoned_at`, RTLM3a1, and +the `derived_from` of the `*CreateWithObjectId` payloads, RTLMV4j5 and RTLCV4g5) are +never encoded. + +A ProtocolMessage's `state` is decoded one `ObjectMessage` at a time: a message that fails +to decode is logged and left out, and the others are decoded. +""" + +from __future__ import annotations + +import base64 +import builtins +import json +import logging +from dataclasses import dataclass, field +from enum import IntEnum +from typing import Any, TypeVar + +log = logging.getLogger(__name__) + +WIRE_FORMAT_JSON = 'json' +WIRE_FORMAT_MSGPACK = 'msgpack' + +E = TypeVar('E', bound='_WireEnum') + + +class _WireEnum(IntEnum): + """An enumeration carried as its integer wire value, whose `UNKNOWN` member stands for a wire + value this library does not recognise.""" + + @classmethod + def from_wire(cls: type[E], value: Any) -> E: + """The member for a wire value, or `UNKNOWN` for one that is not recognised (OOP2a, OMP2a).""" + if isinstance(value, bool): + return cls.UNKNOWN + try: + return cls(value) + except ValueError: + return cls.UNKNOWN + + +class ObjectOperationAction(_WireEnum): + """OOP2: the operation an `ObjectOperation` describes, by its wire value. + + `UNKNOWN` stands for a wire value this library does not recognise (OOP2a). Such an + operation decodes without error, is never applied and is never encoded. + """ + + UNKNOWN = -1 + MAP_CREATE = 0 + MAP_SET = 1 + MAP_REMOVE = 2 + COUNTER_CREATE = 3 + COUNTER_INC = 4 + OBJECT_DELETE = 5 + MAP_CLEAR = 6 + + +class ObjectsMapSemantics(_WireEnum): + """OMP2: the conflict-resolution semantics of a map, by its wire value. + + `UNKNOWN` stands for a wire value this library does not recognise (OMP2a). + """ + + UNKNOWN = -1 + LWW = 0 + + +def _check_format(format: str) -> None: + if format not in (WIRE_FORMAT_JSON, WIRE_FORMAT_MSGPACK): + raise ValueError(f"Unknown wire format {format!r}; expected 'json' or 'msgpack'") + + +def _decode_number(value: Any) -> float | None: + # OD4c3: numbers are float64 on the wire, so a decoded number is always a float + if value is None: + return None + # A value of any other type fails the decode, so from_protocol_message skips its message + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise ValueError(f'Expected a number, got {value!r}') + return float(value) + + +def _encode_number(value: Any, format: str) -> Any: + if format == WIRE_FORMAT_MSGPACK and isinstance(value, int) and not isinstance(value, bool): + return float(value) + return value + + +def _decode_bytes(value: Any) -> Any: + # OD5a1, OD5b2: raw binary on msgpack, a base64 string on JSON + if isinstance(value, (bytes, bytearray)): + return bytes(value) + if isinstance(value, str): + return base64.b64decode(value) + return value + + +def _encode_bytes(value: Any, format: str) -> Any: + # OD4c2, OD4d2 + if format == WIRE_FORMAT_MSGPACK: + return bytes(value) + return base64.b64encode(value).decode('ascii') + + +def _encode_json(value: Any) -> str: + return json.dumps(value, separators=(',', ':')) + + +@dataclass +class ObjectData: + """OD1: a value held in an object, either a primitive or a reference to another object. + + At most one of the value fields is set (OD2). + """ + + object_id: str | None = None # OD2a + encoding: str | None = None # OD2b + boolean: bool | None = None # OD2c + # Spelled `builtins.bytes`, as the field's own name hides the builtin within the class + bytes: builtins.bytes | None = None # OD2d + number: float | None = None # OD2e + string: str | None = None # OD2f + json: dict | list | None = None # OD2g + + @property + def value(self) -> Any: + """The primitive this data holds, or None when it holds a reference or nothing.""" + for candidate in (self.boolean, self.bytes, self.number, self.string, self.json): + if candidate is not None: + return candidate + return None + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> ObjectData: + """Decodes an `ObjectData` per OD5.""" + _check_format(format) + json_value = obj.get('json') + if isinstance(json_value, str): + json_value = json.loads(json_value) + string_value = obj.get('string') + encoding = obj.get('encoding') + # OD5a2, OD5b3: a JSON payload carried in `string` under the legacy `json` encoding + if encoding == 'json' and string_value is not None and json_value is None: + json_value = json.loads(string_value) + string_value = None + encoding = None + return ObjectData( + object_id=obj.get('objectId'), + encoding=encoding, + boolean=obj.get('boolean'), + bytes=_decode_bytes(obj.get('bytes')), + number=_decode_number(obj.get('number')), + string=string_value, + json=json_value, + ) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + """Encodes this `ObjectData` per OD4.""" + _check_format(format) + result: dict = {} + if self.object_id is not None: + result['objectId'] = self.object_id + if self.encoding is not None: + result['encoding'] = self.encoding + if self.boolean is not None: + result['boolean'] = self.boolean + if self.bytes is not None: + result['bytes'] = _encode_bytes(self.bytes, format) + if self.number is not None: + result['number'] = _encode_number(self.number, format) + if self.string is not None: + result['string'] = self.string + if self.json is not None: + result['json'] = _encode_json(self.json) + return result + + +@dataclass +class ObjectsMapEntry: + """OME1: the value at one key of a map. + + `tombstoned_at` (RTLM3a1) is held locally by an `InternalLiveMap` and is never sent + or received. + """ + + tombstone: bool = False # OME2a + timeserial: str | None = None # OME2b + serial_timestamp: int | None = None # OME2d + data: ObjectData | None = None # OME2c + tombstoned_at: int | None = None # RTLM3a1 + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> ObjectsMapEntry: + _check_format(format) + data = obj.get('data') + return ObjectsMapEntry( + tombstone=bool(obj.get('tombstone', False)), + timeserial=obj.get('timeserial'), + serial_timestamp=obj.get('serialTimestamp'), + data=ObjectData.from_dict(data, format) if data is not None else None, + ) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + result: dict = {} + if self.tombstone: + result['tombstone'] = True + if self.timeserial is not None: + result['timeserial'] = self.timeserial + if self.serial_timestamp is not None: + result['serialTimestamp'] = self.serial_timestamp + if self.data is not None: + result['data'] = self.data.to_dict(format) + return result + + +def _decode_entries(entries: dict | None, format: str) -> dict[str, ObjectsMapEntry] | None: + if entries is None: + return None + return {key: ObjectsMapEntry.from_dict(entry or {}, format) for key, entry in entries.items()} + + +def _encode_entries(entries: dict[str, ObjectsMapEntry], format: str) -> dict: + return {key: entry.to_dict(format) for key, entry in entries.items()} + + +@dataclass +class ObjectsMap: + """OMP1: the state of a map object, as an `ObjectState` carries it.""" + + semantics: ObjectsMapSemantics | None = None # OMP3a + entries: dict[str, ObjectsMapEntry] | None = None # OMP3b + clear_timeserial: str | None = None # OMP3c + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> ObjectsMap: + _check_format(format) + semantics = obj.get('semantics') + return ObjectsMap( + semantics=ObjectsMapSemantics.from_wire(semantics) if semantics is not None else None, + entries=_decode_entries(obj.get('entries'), format), + clear_timeserial=obj.get('clearTimeserial'), + ) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + result: dict = {} + if self.semantics is not None: + result['semantics'] = int(self.semantics) + if self.entries is not None: + result['entries'] = _encode_entries(self.entries, format) + if self.clear_timeserial is not None: + result['clearTimeserial'] = self.clear_timeserial + return result + + +@dataclass +class ObjectsCounter: + """OCN1: the state of a counter object, as an `ObjectState` carries it.""" + + count: float | None = None # OCN2a + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> ObjectsCounter: + _check_format(format) + return ObjectsCounter(count=_decode_number(obj.get('count'))) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + return {} if self.count is None else {'count': _encode_number(self.count, format)} + + +@dataclass +class MapCreate: + """MCR1: the payload of a MAP_CREATE operation.""" + + semantics: ObjectsMapSemantics = ObjectsMapSemantics.LWW # MCR2a + entries: dict[str, ObjectsMapEntry] = field(default_factory=dict) # MCR2b + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> MapCreate: + _check_format(format) + semantics = obj.get('semantics') + return MapCreate( + semantics=(ObjectsMapSemantics.from_wire(semantics) if semantics is not None + else ObjectsMapSemantics.LWW), + entries=_decode_entries(obj.get('entries'), format) or {}, + ) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + return {'semantics': int(self.semantics), 'entries': _encode_entries(self.entries, format)} + + +@dataclass +class MapSet: + """MST1: the payload of a MAP_SET operation.""" + + key: str | None = None # MST2a + value: ObjectData | None = None # MST2b + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> MapSet: + _check_format(format) + value = obj.get('value') + return MapSet(key=obj.get('key'), value=ObjectData.from_dict(value, format) if value is not None else None) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + result: dict = {} + if self.key is not None: + result['key'] = self.key + if self.value is not None: + result['value'] = self.value.to_dict(format) + return result + + +@dataclass +class MapRemove: + """MRM1: the payload of a MAP_REMOVE operation.""" + + key: str | None = None # MRM2a + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> MapRemove: + _check_format(format) + return MapRemove(key=obj.get('key')) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + return {} if self.key is None else {'key': self.key} + + +@dataclass +class CounterCreate: + """CCR1: the payload of a COUNTER_CREATE operation.""" + + count: float | None = None # CCR2a + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> CounterCreate: + _check_format(format) + return CounterCreate(count=_decode_number(obj.get('count'))) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + return {} if self.count is None else {'count': _encode_number(self.count, format)} + + +@dataclass +class CounterInc: + """CIN1: the payload of a COUNTER_INC operation.""" + + number: float | None = None # CIN2a + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> CounterInc: + _check_format(format) + return CounterInc(number=_decode_number(obj.get('number'))) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + return {} if self.number is None else {'number': _encode_number(self.number, format)} + + +@dataclass +class ObjectDelete: + """ODE1: the payload of an OBJECT_DELETE operation, which has no attributes (ODE2).""" + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> ObjectDelete: + _check_format(format) + return ObjectDelete() + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + return {} + + +@dataclass +class MapClear: + """MCL1: the payload of a MAP_CLEAR operation, which has no attributes (MCL2).""" + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> MapClear: + _check_format(format) + return MapClear() + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + return {} + + +@dataclass +class MapCreateWithObjectId: + """MCRO1: the payload of a MAP_CREATE operation sent with a client-generated object id. + + `derived_from` is the `MapCreate` the payload was built from, kept for local + application and size calculation (RTLMV4j5) and never sent. + """ + + initial_value: str | None = None # MCRO2a + nonce: str | None = None # MCRO2b + derived_from: MapCreate | None = None # RTLMV4j5 + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> MapCreateWithObjectId: + _check_format(format) + return MapCreateWithObjectId(initial_value=obj.get('initialValue'), nonce=obj.get('nonce')) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + result: dict = {} + if self.initial_value is not None: + result['initialValue'] = self.initial_value + if self.nonce is not None: + result['nonce'] = self.nonce + return result + + +@dataclass +class CounterCreateWithObjectId: + """CCRO1: the payload of a COUNTER_CREATE operation sent with a client-generated object id. + + `derived_from` is the `CounterCreate` the payload was built from, kept for local + application and size calculation (RTLCV4g5) and never sent. + """ + + initial_value: str | None = None # CCRO2a + nonce: str | None = None # CCRO2b + derived_from: CounterCreate | None = None # RTLCV4g5 + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> CounterCreateWithObjectId: + _check_format(format) + return CounterCreateWithObjectId(initial_value=obj.get('initialValue'), nonce=obj.get('nonce')) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + result: dict = {} + if self.initial_value is not None: + result['initialValue'] = self.initial_value + if self.nonce is not None: + result['nonce'] = self.nonce + return result + + +# The operation payloads, by the wire name each is carried under +_PAYLOADS = ( + ('map_create', 'mapCreate', MapCreate), + ('map_set', 'mapSet', MapSet), + ('map_remove', 'mapRemove', MapRemove), + ('counter_create', 'counterCreate', CounterCreate), + ('counter_inc', 'counterInc', CounterInc), + ('object_delete', 'objectDelete', ObjectDelete), + ('map_create_with_object_id', 'mapCreateWithObjectId', MapCreateWithObjectId), + ('counter_create_with_object_id', 'counterCreateWithObjectId', CounterCreateWithObjectId), + ('map_clear', 'mapClear', MapClear), +) + + +@dataclass +class ObjectOperation: + """OOP1: an operation to apply to an object on a channel.""" + + action: ObjectOperationAction # OOP3a + object_id: str # OOP3b + map_create: MapCreate | None = None # OOP3j + map_set: MapSet | None = None # OOP3k + map_remove: MapRemove | None = None # OOP3l + counter_create: CounterCreate | None = None # OOP3m + counter_inc: CounterInc | None = None # OOP3n + object_delete: ObjectDelete | None = None # OOP3o + map_create_with_object_id: MapCreateWithObjectId | None = None # OOP3p + counter_create_with_object_id: CounterCreateWithObjectId | None = None # OOP3q + map_clear: MapClear | None = None # OOP3r + + @property + def resolved_map_create(self) -> MapCreate | None: + """`map_create` if present, else the `MapCreate` `map_create_with_object_id` was derived from. + + This is the `mapCreate` RTLM23 merges and PAOOP3b exposes. + """ + if self.map_create is not None: + return self.map_create + if self.map_create_with_object_id is not None: + return self.map_create_with_object_id.derived_from + return None + + @property + def resolved_counter_create(self) -> CounterCreate | None: + """`counter_create` if present, else the `CounterCreate` `counter_create_with_object_id` was + derived from. + + This is the `counterCreate` RTLC16 merges and PAOOP3c exposes. + """ + if self.counter_create is not None: + return self.counter_create + if self.counter_create_with_object_id is not None: + return self.counter_create_with_object_id.derived_from + return None + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> ObjectOperation: + _check_format(format) + operation = ObjectOperation( + action=ObjectOperationAction.from_wire(obj.get('action')), + object_id=obj.get('objectId'), + ) + for attribute, wire_name, payload_type in _PAYLOADS: + payload = obj.get(wire_name) + if payload is not None: + setattr(operation, attribute, payload_type.from_dict(payload, format)) + return operation + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + if self.action is ObjectOperationAction.UNKNOWN: + # OOP2a: an unrecognised action is never sent + raise ValueError('An ObjectOperation with an unrecognised action cannot be encoded') + result: dict = {'action': int(self.action)} + if self.object_id is not None: + result['objectId'] = self.object_id + for attribute, wire_name, _ in _PAYLOADS: + payload = getattr(self, attribute) + if payload is not None: + result[wire_name] = payload.to_dict(format) + return result + + +@dataclass +class ObjectState: + """OST1: the instantaneous state of an object, as an OBJECT_SYNC carries it.""" + + object_id: str # OST2a + site_timeserials: dict[str, str] = field(default_factory=dict) # OST2b + tombstone: bool = False # OST2c + create_op: ObjectOperation | None = None # OST2d + map: ObjectsMap | None = None # OST2e + counter: ObjectsCounter | None = None # OST2f + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> ObjectState: + _check_format(format) + create_op = obj.get('createOp') + objects_map = obj.get('map') + counter = obj.get('counter') + return ObjectState( + object_id=obj.get('objectId'), + site_timeserials=dict(obj.get('siteTimeserials') or {}), + tombstone=bool(obj.get('tombstone', False)), + create_op=ObjectOperation.from_dict(create_op, format) if create_op is not None else None, + map=ObjectsMap.from_dict(objects_map, format) if objects_map is not None else None, + counter=ObjectsCounter.from_dict(counter, format) if counter is not None else None, + ) + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + _check_format(format) + result: dict = {'objectId': self.object_id, 'siteTimeserials': dict(self.site_timeserials)} + if self.tombstone: + result['tombstone'] = True + if self.create_op is not None: + result['createOp'] = self.create_op.to_dict(format) + if self.map is not None: + result['map'] = self.map.to_dict(format) + if self.counter is not None: + result['counter'] = self.counter.to_dict(format) + return result + + +@dataclass +class ObjectMessage: + """OM1: an object message sent or received over a channel. + + This is the internal type. The type delivered to listeners is the public + `ably.pubsub.liveobjects.publicmessage.ObjectMessage` (PAOM1). + """ + + id: str | None = None # OM2a + client_id: str | None = None # OM2b + connection_id: str | None = None # OM2c + extras: dict | None = None # OM2d + timestamp: int | None = None # OM2e + operation: ObjectOperation | None = None # OM2f + object: ObjectState | None = None # OM2g + serial: str | None = None # OM2h + serial_timestamp: int | None = None # OM2j + site_code: str | None = None # OM2i + + @staticmethod + def from_dict(obj: dict, format: str = WIRE_FORMAT_JSON) -> ObjectMessage: + """Decodes one entry of a ProtocolMessage's `state` array (OM5).""" + _check_format(format) + operation = obj.get('operation') + state = obj.get('object') + return ObjectMessage( + id=obj.get('id'), + client_id=obj.get('clientId'), + connection_id=obj.get('connectionId'), + extras=obj.get('extras'), + timestamp=obj.get('timestamp'), + operation=ObjectOperation.from_dict(operation, format) if operation is not None else None, + object=ObjectState.from_dict(state, format) if state is not None else None, + serial=obj.get('serial'), + serial_timestamp=obj.get('serialTimestamp'), + site_code=obj.get('siteCode'), + ) + + @staticmethod + def from_protocol_message(protocol_message: dict, format: str = WIRE_FORMAT_JSON) -> list[ObjectMessage]: + """Decodes the `state` array of an OBJECT or OBJECT_SYNC ProtocolMessage. + + A message with no `id`, `connectionId` or `timestamp` takes them from the + ProtocolMessage that carried it (OM2a, OM2c, OM2e). A message that fails to decode is + logged and left out, so that it costs neither the other messages nor the handling of + the ProtocolMessage itself, such as the sync cursor of an OBJECT_SYNC. + """ + channel = protocol_message.get('channel') + entries = protocol_message.get('state') or [] + if not isinstance(entries, list): + log.error(f'ObjectMessage.from_protocol_message(): skipping a state that is not an array; ' + f'channel={channel}, state={entries!r}') + return [] + + messages = [] + protocol_id = protocol_message.get('id') + for index, entry in enumerate(entries): + try: + message = ObjectMessage.from_dict(entry, format) + except Exception as e: + log.error(f'ObjectMessage.from_protocol_message(): skipping an object message that failed to ' + f'decode; index={index}, channel={channel}, error={e!r}') + continue + if message.id is None and protocol_id is not None: + message.id = f'{protocol_id}:{index}' + if message.connection_id is None: + message.connection_id = protocol_message.get('connectionId') + if message.timestamp is None: + message.timestamp = protocol_message.get('timestamp') + messages.append(message) + return messages + + def to_dict(self, format: str = WIRE_FORMAT_JSON) -> dict: + """Encodes this message for the `state` array of a ProtocolMessage (OM4).""" + _check_format(format) + result: dict = {} + for attribute, wire_name in (('id', 'id'), ('client_id', 'clientId'), ('connection_id', 'connectionId'), + ('extras', 'extras'), ('timestamp', 'timestamp'), ('serial', 'serial'), + ('serial_timestamp', 'serialTimestamp'), ('site_code', 'siteCode')): + value = getattr(self, attribute) + if value is not None: + result[wire_name] = value + if self.operation is not None: + result['operation'] = self.operation.to_dict(format) + if self.object is not None: + result['object'] = self.object.to_dict(format) + return result diff --git a/ably/pubsub/liveobjects/objectspool.py b/ably/pubsub/liveobjects/objectspool.py new file mode 100644 index 00000000..741b120b --- /dev/null +++ b/ably/pubsub/liveobjects/objectspool.py @@ -0,0 +1,114 @@ +"""The objects held for a channel (RTO3).""" + +from __future__ import annotations + +import logging +from collections.abc import Iterator, MutableMapping +from typing import TYPE_CHECKING + +from ably.pubsub.liveobjects.defaults import ROOT_OBJECT_ID +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.livemap import InternalLiveMap +from ably.pubsub.liveobjects.liveobject import LiveObject +from ably.pubsub.liveobjects.objectid import object_type_of +from ably.pubsub.util.clock import Clock + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.realtimeobject import RealtimeObject + +log = logging.getLogger(__name__) + + +class ObjectsPool(MutableMapping): + """RTO3: the live objects on a channel, by object id (RTO3a). + + A pool always holds an `InternalLiveMap` with id `root`, created with the pool + (RTO3b1). It is a mutable mapping, so `pool[object_id]`, `object_id in pool`, + `len(pool)` and `pool.keys()` work as they do on a dict. Setting `pool[object_id] = obj` + adopts `obj`: its `pool` becomes this pool. + + A pool is constructible on its own. `realtime_object` is None for a pool built without + one, in which case objects in it dispatch no path subscriptions and publish nothing. + """ + + def __init__(self, realtime_object: RealtimeObject | None = None, *, clock: Clock | None = None): + self.realtime_object: RealtimeObject | None = realtime_object + self.clock: Clock = clock if clock is not None else Clock() + self._objects: dict[str, LiveObject] = {} + self[ROOT_OBJECT_ID] = InternalLiveMap(ROOT_OBJECT_ID, pool=self) + + @property + def root(self) -> InternalLiveMap: + """The root map (RTO3b).""" + return self._objects[ROOT_OBJECT_ID] + + def __getitem__(self, object_id: str) -> LiveObject: + return self._objects[object_id] + + def __setitem__(self, object_id: str, live_object: LiveObject) -> None: + live_object.pool = self + self._objects[object_id] = live_object + + def __delitem__(self, object_id: str) -> None: + del self._objects[object_id] + + def __iter__(self) -> Iterator[str]: + return iter(self._objects) + + def __len__(self) -> int: + return len(self._objects) + + def __repr__(self) -> str: + return f'ObjectsPool({list(self._objects)!r})' + + def create_zero_value_object_if_not_exists(self, object_id: str) -> LiveObject | None: + """RTO6: the object with `object_id`, created empty from the type its id names if absent. + + Returns None, creating nothing, for an id that names neither a map nor a counter. + """ + existing = self._objects.get(object_id) + if existing is not None: + return existing # RTO6a + + object_type = object_type_of(object_id) if isinstance(object_id, str) else None # RTO6b1 + if object_type == 'map': + live_object: LiveObject = InternalLiveMap(object_id) # RTO6b2 + elif object_type == 'counter': + live_object = InternalLiveCounter(object_id) # RTO6b3 + else: + log.warning(f'ObjectsPool.create_zero_value_object_if_not_exists(): cannot create an object for ' + f'an id of unknown type; object_id={object_id!r}') + return None + + self[object_id] = live_object + return live_object + + def clear_all_data(self) -> None: + """RTO27a1: resets every object's data to empty, keeping the objects, emitting nothing.""" + for live_object in list(self._objects.values()): + live_object.clear_data() + + def rebuild_parent_references(self) -> None: + """RTO5c10: clears every object's `parent_references` and rebuilds them from the map entries.""" + for live_object in self._objects.values(): + live_object.parent_references = {} # RTO5c10a + + # RTO5c10b + for live_object in list(self._objects.values()): + if not isinstance(live_object, InternalLiveMap): + continue + for key, value in live_object.entries(): + if isinstance(value, LiveObject): + value.add_parent_reference(live_object, key) + + def collect_garbage(self, grace_period_ms: int, now_ms: int) -> None: + """RTO10c: releases tombstoned map entries (RTLM19) and objects tombstoned `grace_period_ms` + or more before `now_ms`, never the root (RTO10c1b1).""" + for object_id, live_object in list(self._objects.items()): + # RTO10c1b, RTO10c1b1 + if (object_id != ROOT_OBJECT_ID and live_object.is_tombstone and live_object.tombstoned_at is not None + and now_ms - live_object.tombstoned_at >= grace_period_ms): + del self._objects[object_id] + continue + if isinstance(live_object, InternalLiveMap): + live_object.gc_tombstoned_entries(grace_period_ms, now_ms) # RTO10c1a diff --git a/ably/pubsub/liveobjects/pathobject.py b/ably/pubsub/liveobjects/pathobject.py new file mode 100644 index 00000000..2a8d15ec --- /dev/null +++ b/ably/pubsub/liveobjects/pathobject.py @@ -0,0 +1,317 @@ +"""Path-addressed views onto a channel's objects (RTPO*, RTTS3-RTTS6). + +A `PathObject` holds a path from the root map and resolves it each time a method reads +or writes through it, so it follows whatever object is at that path at the time. The +base class carries what does not depend on the type at the path, navigation included; +`as_live_map()`, `as_live_counter()` and `as_primitive()` return the typed views, without +checking what the path resolves to (RTTS5d). + +Reads never raise for a path that does not resolve or that resolves to another type: +they return None, or an empty list for the collection methods (RTPO3c1, RTTS5d1). Writes +raise `AblyException` 92005 for a path that does not resolve and 92007 for one that +resolves to the wrong type (RTPO3c2, RTTS5d2). Every read and `subscribe` checks the +access preconditions (RTO25), and every write the write preconditions (RTO26), before +resolving the path; navigation and the view helpers check nothing. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, Callable, overload + +from ably.pubsub.liveobjects.instance import ( + Instance, + compact_json_value, + expected_value_type, + primitive_value, + value_type_of, +) +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.livemap import InternalLiveMap +from ably.pubsub.liveobjects.publicmessage import ObjectMessage +from ably.pubsub.util.exceptions import AblyException + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.batch import Batch, LiveCounterBatchContext, LiveMapBatchContext + from ably.pubsub.liveobjects.enums import ValueType + from ably.pubsub.liveobjects.realtimeobject import RealtimeObject + from ably.pubsub.liveobjects.subscription import Subscription + from ably.pubsub.liveobjects.valuetypes import JsonValue, LiveMapValue, Primitive, T + + +@dataclass +class PathObjectSubscriptionEvent: + """RTPO19e: what a `PathObject.subscribe` listener receives.""" + + object: PathObject # RTPO19e1: the path where the change occurred + message: ObjectMessage | None = None # RTPO19e2 + + +class PathObject: + """RTPO1, RTTS3: a lazy reference to whatever is at a path from the root map.""" + + def __init__(self, realtime_object: RealtimeObject, root: InternalLiveMap, path: Sequence[str]): + self._realtime_object = realtime_object + self._root = root # RTPO2b + self._path: list[str] = list(path) # RTPO2a + + def __repr__(self) -> str: + return f'{type(self).__name__}({self.path()!r})' + + def path(self) -> str: + """RTPO4: the path as a dotted string, with dots in segments escaped as `\\.`.""" + # RTPO4a, RTPO4b; the empty path is the empty string (RTPO4c) + return '.'.join(segment.replace('.', '\\.') for segment in self._path) + + def type(self) -> ValueType | None: + """RTTS4b: the type of what the path resolves to, or None if it resolves to nothing.""" + self._realtime_object._check_access_preconditions() # RTTS4b1 + resolved = self._resolve() # RTTS4b2 + return value_type_of(resolved) if resolved is not None else None # RTTS4b3 + + def exists(self) -> bool: + """RTTS4a: whether the path resolves to anything.""" + self._realtime_object._check_access_preconditions() # RTTS4a1 + return self._resolve() is not None # RTTS4a2, RTTS4a3 + + def get(self, key: str) -> PathObject: + """RTPO5: the path one key further down. Navigational only (RTPO5d). + + Raises AblyException 40003 if `key` is not a string (RTPO5b). + """ + if not isinstance(key, str): + raise AblyException(f'Path key must be a string, not {type(key).__name__}', 400, 40003) + return PathObject(self._realtime_object, self._root, [*self._path, key]) # RTPO5c + + def at(self, path: str | Sequence[str]) -> PathObject: + """RTPO6: the path further down by a dotted string, `\\.` escaping a literal dot (RTPO6b), + or by a sequence of segments taken as they are. + + Raises AblyException 40003 for anything else. + """ + if isinstance(path, str): + segments = _parse_path(path) # RTPO6b + elif (isinstance(path, Sequence) and not isinstance(path, (bytes, bytearray)) + and all(isinstance(segment, str) for segment in path)): + segments = list(path) + else: + raise AblyException(f'Path must be a string or a sequence of strings, not {type(path).__name__}', + 400, 40003) + return PathObject(self._realtime_object, self._root, [*self._path, *segments]) # RTPO6c + + def instance(self) -> Instance | None: + """RTPO8: an `Instance` wrapping what the path resolves to, or None if it resolves to nothing. + + The instance is the subclass matching the resolved value (`LiveMapInstance`, + `LiveCounterInstance` or `PrimitiveInstance`). + """ + self._realtime_object._check_access_preconditions() # RTPO8a + resolved = self._resolve() # RTPO8b + if resolved is None: + return None # RTPO8e + return Instance._wrap(self._realtime_object, resolved) # RTPO8c, RTPO8f + + def compact_json(self) -> JsonValue: + """RTPO14: a JSON-serializable snapshot of what the path resolves to, or None. + + A map is a JSON object of its entries that are not tombstoned, nested maps becoming + nested objects and counters their values. Binary is a base64 string, and a map + reached again in the same snapshot, through a cycle or a second reference, is + `{'objectId': id}`. + """ + self._realtime_object._check_access_preconditions() # RTPO14a + resolved = self._resolve() # RTPO13b + if resolved is None: + return None # RTPO13f + return compact_json_value(resolved) # RTPO14b + + def subscribe(self, listener: Callable[[PathObjectSubscriptionEvent], None], *, + depth: int | None = None) -> Subscription: + """RTPO19: calls `listener` for changes at this path or, within `depth` levels, below it. + + Raises AblyException 40003 if `depth` is given and is not a positive integer + (RTPO19c1a). Has no effect on the channel (RTPO19g). + """ + self._realtime_object._check_access_preconditions() # RTPO19b + if depth is not None and (isinstance(depth, bool) or not isinstance(depth, int) or depth < 1): + # RTPO19c1a + raise AblyException(f'Subscription depth must be a positive integer, or None for any depth; ' + f'got {depth!r}', 400, 40003) + register = self._realtime_object._path_object_subscription_register + return register.subscribe(self._path, listener, depth) # RTPO19f + + def as_live_map(self) -> LiveMapPathObject: + """RTTS5a: this path, viewed as a map.""" + return LiveMapPathObject(self._realtime_object, self._root, self._path) + + def as_live_counter(self) -> LiveCounterPathObject: + """RTTS5b: this path, viewed as a counter.""" + return LiveCounterPathObject(self._realtime_object, self._root, self._path) + + def as_primitive(self) -> PrimitivePathObject: + """RTTS5c, RTTS6h: this path, viewed as a primitive.""" + return PrimitivePathObject(self._realtime_object, self._root, self._path) + + def _resolve(self) -> Any: + """RTPO3: what the path resolves to, or None if it does not resolve (RTPO3c).""" + current: Any = self._root # RTPO3b + for segment in self._path: # RTPO3a + if not isinstance(current, InternalLiveMap): + return None # RTPO3a1 + current = current.get(segment) # RTPO3a2, RTPO3a3 + if current is None: + return None + return current + + def _resolve_for_write(self, expected_type: type, description: str) -> Any: + """RTPO3c2, RTTS5d2: what the path resolves to, for a write that requires an `expected_type`. + + Raises AblyException 92005 if the path does not resolve and 92007 if it resolves to + something else. + """ + resolved = self._resolve() + if resolved is None: + raise AblyException(f'Could not resolve a value at path {self.path()!r}', 400, 92005) + if not isinstance(resolved, expected_type): + raise AblyException(f'Cannot write to the value at path {self.path()!r} as {description}: it is of ' + f'type {value_type_of(resolved).value}', 400, 92007) + return resolved + + +class LiveMapPathObject(PathObject): + """RTTS6a: a path expected to resolve to a map.""" + + def batch(self) -> Batch[LiveMapBatchContext]: + """RTPO20: a block whose queued writes are published as one message when it exits. + + Entering checks the write preconditions (RTPO20b) and raises AblyException 92007 if + the path does not resolve to a map (RTPO20c). + """ + # Imported here, as the batch module builds on the views + from ably.pubsub.liveobjects.batch import Batch, LiveMapBatchContext + + return Batch(self._realtime_object, self._resolve, LiveMapBatchContext, f'path {self.path()!r}') + + def entries(self) -> list[tuple[str, PathObject]]: + """RTPO9: `(key, path)` for each key of the map, or `[]`.""" + self._realtime_object._check_access_preconditions() # RTPO9a + resolved = self._resolve() # RTPO9b + if not isinstance(resolved, InternalLiveMap): + return [] # RTPO9d + return [(key, self.get(key)) for key in resolved.keys()] # RTPO9c + + def keys(self) -> list[str]: + """RTPO10: the keys of the map, or `[]`.""" + self._realtime_object._check_access_preconditions() # RTPO10a + resolved = self._resolve() # RTPO10b + if not isinstance(resolved, InternalLiveMap): + return [] # RTPO10d + return resolved.keys() # RTPO10c + + def values(self) -> list[PathObject]: + """RTPO11: a path for each key of the map, or `[]`.""" + return [path_object for _, path_object in self.entries()] # RTPO11a-RTPO11d + + def size(self) -> int | None: + """RTPO12: the number of entries in the map, or None.""" + self._realtime_object._check_access_preconditions() # RTPO12a + resolved = self._resolve() # RTPO12b + if not isinstance(resolved, InternalLiveMap): + return None # RTPO12d + return resolved.size() # RTPO12c + + async def set(self, key: str, value: LiveMapValue) -> None: + """RTPO15: sets `key` in the map at this path.""" + self._realtime_object._check_write_preconditions() # RTPO15b + live_map = self._resolve_for_write(InternalLiveMap, 'a map') # RTPO15c, RTPO15e + await live_map.set(key, value) # RTPO15d + + async def remove(self, key: str) -> None: + """RTPO16: removes `key` from the map at this path.""" + self._realtime_object._check_write_preconditions() # RTPO16b + live_map = self._resolve_for_write(InternalLiveMap, 'a map') # RTPO16c, RTPO16e + await live_map.remove(key) # RTPO16d + + +class LiveCounterPathObject(PathObject): + """RTTS6b: a path expected to resolve to a counter.""" + + def batch(self) -> Batch[LiveCounterBatchContext]: + """RTPO20: a block whose queued writes are published as one message when it exits. + + Entering checks the write preconditions (RTPO20b) and raises AblyException 92007 if + the path does not resolve to a counter (RTPO20c). + """ + # Imported here, as the batch module builds on the views + from ably.pubsub.liveobjects.batch import Batch, LiveCounterBatchContext + + return Batch(self._realtime_object, self._resolve, LiveCounterBatchContext, f'path {self.path()!r}') + + def value(self) -> float | None: + """RTTS6b: the counter's value, or None if the path does not resolve to a counter.""" + self._realtime_object._check_access_preconditions() # RTPO7a + resolved = self._resolve() # RTPO7b + if not isinstance(resolved, InternalLiveCounter): + return None # RTPO7f, RTTS6b + return resolved.value() # RTPO7c + + async def increment(self, amount: float = 1) -> None: + """RTPO17: increments the counter at this path by `amount`.""" + self._realtime_object._check_write_preconditions() # RTPO17b + counter = self._resolve_for_write(InternalLiveCounter, 'a counter') # RTPO17c, RTPO17e + await counter.increment(amount) # RTPO17d + + async def decrement(self, amount: float = 1) -> None: + """RTPO18: decrements the counter at this path by `amount`.""" + self._realtime_object._check_write_preconditions() # RTPO18b + counter = self._resolve_for_write(InternalLiveCounter, 'a counter') # RTPO18c, RTPO18e + await counter.decrement(amount) # RTPO18d + + +class PrimitivePathObject(PathObject): + """RTTS6c, RTTS6h: a path expected to resolve to a primitive.""" + + @overload + def value(self) -> Primitive | None: ... + + @overload + def value(self, expected: type[T]) -> T | None: ... + + def value(self, expected: type | None = None) -> Any: + """RTPO7, RTTS6c: the primitive at this path, or None. + + With `expected` (one of `str`, `float`, `bool`, `bytes`, `list`, `dict`) the value is + returned only if it is of that type, judged by its wire type, so a boolean is never + a number. Any other `expected` raises TypeError. + """ + value_type = expected_value_type(expected) + self._realtime_object._check_access_preconditions() # RTPO7a + return primitive_value(self._resolve(), value_type) # RTPO7b, RTPO7d-RTPO7f, RTTS6c + + +def _parse_path(path: str) -> list[str]: + """RTPO6b: the segments of a dotted path, in which `\\.` is a literal dot. + + A backslash before anything other than a dot is kept as it is. + """ + segments: list[str] = [] + segment: list[str] = [] + escaping = False + for char in path: + if escaping: + if char != '.': + segment.append('\\') + segment.append(char) + escaping = False + elif char == '\\': + escaping = True + elif char == '.': + segments.append(''.join(segment)) + segment = [] + else: + segment.append(char) + if escaping: + segment.append('\\') + segments.append(''.join(segment)) + return segments diff --git a/ably/pubsub/liveobjects/pathobjectsubscriptionregister.py b/ably/pubsub/liveobjects/pathobjectsubscriptionregister.py new file mode 100644 index 00000000..d78cee9f --- /dev/null +++ b/ably/pubsub/liveobjects/pathobjectsubscriptionregister.py @@ -0,0 +1,89 @@ +"""The register of path subscriptions a channel's `RealtimeObject` keeps (RTO24).""" + +from __future__ import annotations + +import logging +from typing import TYPE_CHECKING, Callable + +from ably.pubsub.liveobjects.liveobject import LiveMapUpdate +from ably.pubsub.liveobjects.pathobject import PathObject, PathObjectSubscriptionEvent +from ably.pubsub.liveobjects.publicmessage import ObjectMessage +from ably.pubsub.liveobjects.subscription import Registry, Subscription + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.liveobject import LiveObject, LiveObjectUpdate + from ably.pubsub.liveobjects.realtimeobject import RealtimeObject + +log = logging.getLogger(__name__) + + +class _PathSubscription: + """One `PathObject.subscribe` registration.""" + + def __init__(self, path: list[str], listener: Callable[[PathObjectSubscriptionEvent], None], + depth: int | None): + self.path = path + self.listener = listener + self.depth = depth + + +class PathObjectSubscriptionRegister: + """RTO24: every subscription made through `PathObject.subscribe` on one channel (RTO24a).""" + + def __init__(self, realtime_object: RealtimeObject): + self.realtime_object = realtime_object + self._subscriptions: Registry[_PathSubscription] = Registry() + + def subscribe(self, path: list[str], listener: Callable[[PathObjectSubscriptionEvent], None], + depth: int | None = None) -> Subscription: + """RTPO19f: registers `listener` for changes covered by `path` and `depth` (RTO24c1).""" + return Subscription(self._subscriptions.register(_PathSubscription(list(path), listener, depth))) + + def dispatch(self, live_object: LiveObject, update: LiveObjectUpdate) -> None: + """RTO24b: calls each subscription once for each path to `live_object` that it covers. + + A listener that raises is logged and does not stop the others (RTO24b2c). A + subscription made during a dispatch is not called by it, for any path, and one removed + during a dispatch is not called again. + """ + if not self._subscriptions: + return + registrations = self._subscriptions.snapshot() + + realtime_object = self.realtime_object + root = realtime_object._objects_pool.root + object_message = update.object_message + # RTO24b2b2: an update from a sync carries the object's state, not an operation + has_operation = object_message is not None and object_message.operation is not None + updated_keys = list(update.update) if isinstance(update, LiveMapUpdate) else [] + + for path_to_this in live_object.get_full_paths(): # RTO24b1, RTO24b2 + # RTO24b2a1, RTO24b2a2 + candidate_paths = [path_to_this, *([*path_to_this, key] for key in updated_keys)] + for registration in registrations: + if not registration.active: + continue + subscription = registration.item + # RTO24b2b + event_path = next((candidate for candidate in candidate_paths + if self.covers(subscription.path, subscription.depth, candidate)), None) + if event_path is None: + continue + try: + message = None + if has_operation: + message = ObjectMessage._from_internal(object_message, realtime_object._channel_name) + event = PathObjectSubscriptionEvent(PathObject(realtime_object, root, event_path), message) + subscription.listener(event) + except Exception: + # RTO24b2c + log.exception(f'PathObjectSubscriptionRegister.dispatch(): a path subscription listener ' + f'raised; path={event_path}, channel={realtime_object._channel_name}') + + @staticmethod + def covers(subscription_path: list[str], depth: int | None, event_path: list[str]) -> bool: + """RTO24c1: whether a subscription at `subscription_path` with `depth` covers `event_path`.""" + prefix_length = len(subscription_path) + if len(event_path) < prefix_length or list(event_path[:prefix_length]) != list(subscription_path): + return False + return depth is None or len(event_path) - prefix_length + 1 <= depth diff --git a/ably/pubsub/liveobjects/publicmessage.py b/ably/pubsub/liveobjects/publicmessage.py new file mode 100644 index 00000000..49642452 --- /dev/null +++ b/ably/pubsub/liveobjects/publicmessage.py @@ -0,0 +1,102 @@ +"""The object message delivered to subscription listeners (PAOM*, PAOOP*). + +The specification calls these `PublicAPI::ObjectMessage` and `PublicAPI::ObjectOperation`. +They are exposed to users as `ObjectMessage` and `ObjectOperation`, sharing their names +with the internal wire types in `ably.pubsub.liveobjects.objectmessage`; code that needs both +imports one of the pairs under a module-qualified name. +""" + +from __future__ import annotations + +import copy +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from ably.pubsub.liveobjects.objectmessage import ( + CounterCreate, + CounterInc, + MapClear, + MapCreate, + MapRemove, + MapSet, + ObjectDelete, + ObjectOperationAction, +) + +if TYPE_CHECKING: + from ably.pubsub.liveobjects import objectmessage + + +@dataclass +class ObjectOperation: + """PAOOP1: the operation that caused an object change. + + Unlike the wire operation it never carries the `*CreateWithObjectId` payloads: a + create sent with a client-generated object id is shown as the `MapCreate` or + `CounterCreate` it was derived from. + """ + + action: ObjectOperationAction # PAOOP2a + object_id: str # PAOOP2b + map_create: MapCreate | None = None # PAOOP2c + map_set: MapSet | None = None # PAOOP2d + map_remove: MapRemove | None = None # PAOOP2e + counter_create: CounterCreate | None = None # PAOOP2f + counter_inc: CounterInc | None = None # PAOOP2g + object_delete: ObjectDelete | None = None # PAOOP2h + map_clear: MapClear | None = None # PAOOP2i + + @staticmethod + def _from_internal(operation: objectmessage.ObjectOperation) -> ObjectOperation: + """PAOOP3: the public form of an internal `ObjectOperation`. + + The payloads are copies, so a listener that changes them leaves the objects the + operation was applied to unchanged. + """ + return ObjectOperation( + action=operation.action, # PAOOP3a + object_id=operation.object_id, + map_create=copy.deepcopy(operation.resolved_map_create), # PAOOP3b + map_set=copy.deepcopy(operation.map_set), + map_remove=copy.deepcopy(operation.map_remove), + counter_create=copy.deepcopy(operation.resolved_counter_create), # PAOOP3c + counter_inc=copy.deepcopy(operation.counter_inc), + object_delete=copy.deepcopy(operation.object_delete), + map_clear=copy.deepcopy(operation.map_clear), + ) + + +@dataclass +class ObjectMessage: + """PAOM1: the inbound object message that caused an object change.""" + + channel: str # PAOM2e + operation: ObjectOperation # PAOM2f + id: str | None = None # PAOM2a + client_id: str | None = None # PAOM2b + connection_id: str | None = None # PAOM2c + timestamp: int | None = None # PAOM2d + serial: str | None = None # PAOM2g + serial_timestamp: int | None = None # PAOM2h + site_code: str | None = None # PAOM2i + extras: dict | None = None # PAOM2j + + @staticmethod + def _from_internal(message: objectmessage.ObjectMessage, channel_name: str) -> ObjectMessage: + """PAOM3: the public form of an internal `ObjectMessage` received on `channel_name`. + + The caller ensures `message.operation` is populated (PAOM3a1). + """ + return ObjectMessage( + channel=channel_name, # PAOM3b + operation=ObjectOperation._from_internal(message.operation), # PAOM3d + # PAOM3c + id=message.id, + client_id=message.client_id, + connection_id=message.connection_id, + timestamp=message.timestamp, + serial=message.serial, + serial_timestamp=message.serial_timestamp, + site_code=message.site_code, + extras=copy.deepcopy(message.extras), + ) diff --git a/ably/pubsub/liveobjects/realtimeobject.py b/ably/pubsub/liveobjects/realtimeobject.py new file mode 100644 index 00000000..62c4792d --- /dev/null +++ b/ably/pubsub/liveobjects/realtimeobject.py @@ -0,0 +1,645 @@ +"""The LiveObjects entry point for a channel (RTO*), reached as `channel.object`.""" + +from __future__ import annotations + +import asyncio +import dataclasses +import logging +from typing import TYPE_CHECKING, Any, Callable + +from ably.pubsub.liveobjects.defaults import GC_GRACE_PERIOD_MS, GC_INTERVAL_MS, ROOT_OBJECT_ID +from ably.pubsub.liveobjects.enums import ObjectsEvent, ObjectsOperationSource, ObjectsSyncState +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.livemap import InternalLiveMap +from ably.pubsub.liveobjects.liveobject import LiveObject, LiveObjectUpdate +from ably.pubsub.liveobjects.objectmessage import ( + WIRE_FORMAT_JSON, + WIRE_FORMAT_MSGPACK, + ObjectOperationAction, + ObjectsMapSemantics, +) +from ably.pubsub.liveobjects.objectspool import ObjectsPool +from ably.pubsub.liveobjects.pathobject import LiveMapPathObject +from ably.pubsub.liveobjects.pathobjectsubscriptionregister import PathObjectSubscriptionRegister +from ably.pubsub.liveobjects.subscription import Registry, StatusSubscription +from ably.pubsub.liveobjects.syncobjectspool import SyncObjectsPool +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.clock import Clock, select_clock +from ably.pubsub.util.exceptions import AblyException + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.objectmessage import ObjectMessage + from ably.pubsub.realtime.channel import RealtimeChannel + from ably.pubsub.types.connectiondetails import ConnectionDetails + from ably.pubsub.types.operations import PublishResult + from ably.pubsub.util.helper import Timer + +log = logging.getLogger(__name__) + +# RTO9a2a: the operation actions `_apply_object_messages` applies +_SUPPORTED_ACTIONS = frozenset(( + ObjectOperationAction.MAP_CREATE, + ObjectOperationAction.MAP_SET, + ObjectOperationAction.MAP_REMOVE, + ObjectOperationAction.COUNTER_CREATE, + ObjectOperationAction.COUNTER_INC, + ObjectOperationAction.OBJECT_DELETE, + ObjectOperationAction.MAP_CLEAR, +)) + +# RTO17b: the event each sync state emits when it is entered +_SYNC_STATE_EVENTS = { + ObjectsSyncState.SYNCING: ObjectsEvent.SYNCING, + ObjectsSyncState.SYNCED: ObjectsEvent.SYNCED, +} + +# RTO23c1, RTO20e1: the channel states that fail a wait for SYNCED +_SYNC_WAIT_FAILURE_STATES = frozenset((ChannelState.DETACHED, ChannelState.SUSPENDED, ChannelState.FAILED)) + +# RTO27a: the channel states after which the objects' data can no longer be known. The channel +# leaves them only by attaching again, so a wait for SYNCED begun in one of them fails at once. +_DATA_CLEARING_STATES = frozenset((ChannelState.DETACHED, ChannelState.FAILED)) + + +class RealtimeObject: + """RTO*: the objects on one channel, and the entry point to them. + + Each `RealtimeChannel` builds one, reachable as `channel.object`. One can also be built + with no channel, so that its sync and apply behaviour can be driven directly; such a + `RealtimeObject` publishes nothing, checks no channel preconditions and schedules no + timers. `pool` adopts an existing `ObjectsPool`; `clock` defaults to the channel's + client's clock. + + Construction is cheap and needs no running event loop: the GC timer (RTO10) is + scheduled on the first ATTACHED, not here. + """ + + def __init__(self, channel: RealtimeChannel | None = None, *, pool: ObjectsPool | None = None, + clock: Clock | None = None): + self._channel = channel + if clock is None: + clock = select_clock(channel.ably.options) if channel is not None else Clock() + self._clock: Clock = clock + if pool is None: + pool = ObjectsPool(self, clock=clock) + else: + pool.realtime_object = self + self._objects_pool: ObjectsPool = pool # RTO3 + self._sync_objects_pool = SyncObjectsPool() # RTO5f + self._sync_state = ObjectsSyncState.INITIALIZED # RTO17a1 + self._buffered_object_operations: list[ObjectMessage] = [] # RTO7a, RTO7a1 + self._applied_on_ack_serials: set[str] = set() # RTO7b, RTO7b1 + self._current_sync_id: str | None = None # RTO5a1 + self._path_object_subscription_register = PathObjectSubscriptionRegister(self) # RTO24a + self._gc_interval_ms: int = GC_INTERVAL_MS # RTO10a, read each time the GC timer is scheduled + self._gc_timer: Timer | None = None + self._gc_grace_period_ms: int = GC_GRACE_PERIOD_MS # RTO10b3, until a CONNECTED gives one + self._server_time_offset_ms: float | None = None # RTO16a + # RTO23c, RTO20e: each pending wait for SYNCED, with the description its 92008 error gives + self._sync_waiters: dict[asyncio.Future[None], str] = {} + # RTO18: the callbacks `on` registered, by event + self._sync_state_listeners: dict[ObjectsEvent, Registry[Callable[[], None]]] = { + event: Registry() for event in ObjectsEvent + } + # Whether the channel has been released, after which it receives nothing more + self._released = False + if channel is not None: + self._on_connected(channel.ably.connection.connection_details) # RTO10b1 + + # Public API + + async def get(self) -> LiveMapPathObject: + """RTO23: the root of the channel's objects, once they have synced. + + Requires the OBJECT_SUBSCRIBE mode (RTO23a, 40024), attaches the channel if it is not + attached (RTO23e, RTL33), and waits for the sync state to reach SYNCED (RTO23c). + Raises AblyException 92008 if the channel is or enters DETACHED or FAILED, or enters + SUSPENDED, while waiting (RTO23c1), 90001 if the channel is FAILED (RTL33c), and the + attach's error if attaching fails (RTO23e). + """ + self._throw_if_missing_channel_mode(ChannelMode.OBJECT_SUBSCRIBE) # RTO23a + if self._channel is not None: + await self._channel._ensure_active() # RTO23e + + if self._sync_state != ObjectsSyncState.SYNCED: + await self._wait_for_synced('The object could not be retrieved') # RTO23c, RTO23c1 + + return LiveMapPathObject(self, self._objects_pool.root, []) # RTO23d + + def on(self, event: ObjectsEvent, callback: Callable[[], None]) -> StatusSubscription: + """RTO18: calls `callback`, with no arguments, whenever the sync state reaches `event`. + + Registering one callback twice calls it twice (RTO18d). Raises AblyException 40003 for + an `event` that is not an `ObjectsEvent`. + """ + return StatusSubscription(self._sync_state_listeners[_objects_event(event)].register(callback)) # RTO18c + + def off(self, event: ObjectsEvent, callback: Callable[[], None]) -> None: + """RTO19: deregisters `callback` from `event`, however many times it was registered. + + Raises AblyException 40003 for an `event` that is not an `ObjectsEvent`. + """ + self._sync_state_listeners[_objects_event(event)].deregister(callback) + + # Internal API: inbound protocol messages and channel state + + @property + def _channel_name(self) -> str | None: + """The channel's name, which PAOM3b puts on every public message.""" + return self._channel.name if self._channel is not None else None + + @property + def _wire_format(self) -> str: + """The wire format the channel's connection decodes with.""" + if self._channel is not None and self._channel.ably.options.use_binary_protocol: + return WIRE_FORMAT_MSGPACK + return WIRE_FORMAT_JSON + + def _on_attached(self, has_objects: bool) -> None: + """RTO4: handles an ATTACHED ProtocolMessage the channel received while ATTACHING or + ATTACHED; the channel ignores one received in any other state. + + `has_objects` is the HAS_OBJECTS flag. Starts the GC timer if it is not running + and there is a channel (RTO10). + """ + log.debug(f'RealtimeObject._on_attached(): channel={self._channel_name}, has_objects={has_objects}') + self._set_sync_state(ObjectsSyncState.SYNCING) # RTO4c + self._buffered_object_operations = [] # RTO4d + # The objects an ATTACHED announces arrive in a sync sequence of their own (RTO4a), so a + # sequence still in flight from before it is abandoned + self._start_sync_sequence(None) + + if not has_objects: + # RTO4b + pool = self._objects_pool + for object_id in list(pool): + if object_id != ROOT_OBJECT_ID: + del pool[object_id] # RTO4b1 + + # RTO4b2: the root is cleared in place, never replaced + root = pool.root + previous_data = root.data + root.clear_data() + # RTO4b2a: the removed keys, with no object message; a no-op if the root was empty + root.notify_updated(InternalLiveMap.diff(previous_data, root.data)) + + self._sync_objects_pool.clear() # RTO4b3 + self._complete_sync() # RTO4b4 + + self._schedule_gc_timer() # RTO10a + + def _on_connected(self, connection_details: ConnectionDetails | None) -> None: + """RTO10b2: takes the GC grace period from the ConnectionDetails of a CONNECTED. + + With no details, as before the first CONNECTED, the grace period is left as it is. + """ + if connection_details is None: + return + grace_period_ms = connection_details.objects_gc_grace_period + # RTO10b1, RTO10b3 + self._gc_grace_period_ms = grace_period_ms if grace_period_ms is not None else GC_GRACE_PERIOD_MS + + def _release(self) -> None: + """Stops this RealtimeObject when its channel is released (RTS4). + + A released channel receives nothing more, so the GC timer stops for good, and every + wait for SYNCED, pending or later, fails with 92008. + """ + self._released = True + self._cancel_gc_timer() + self._fail_sync_waiters('the channel being released', self._channel_error_reason) + + def _handle_object_sync_messages(self, object_messages: list[ObjectMessage], + sync_channel_serial: str | None) -> None: + """RTO5: handles the decoded `state` of an OBJECT_SYNC ProtocolMessage.""" + sync_id, sync_cursor = self._parse_sync_channel_serial(sync_channel_serial) # RTO5a + self._set_sync_state(ObjectsSyncState.SYNCING) # RTO5e + + # RTO5a2: a new sequence id starts a new sequence. A message with no sequence id holds a + # whole sync of its own (RTO5a5), so it too discards anything accumulated before it. + if sync_id is None or sync_id != self._current_sync_id: + self._start_sync_sequence(sync_id) + + self._sync_objects_pool.apply_object_sync_messages(object_messages) # RTO5f + + # RTO5a4, RTO5a5: the sequence is complete once its cursor is empty + if not sync_cursor: + self._complete_sync() + + def _handle_object_messages(self, object_messages: list[ObjectMessage]) -> None: + """RTO8: handles the decoded `state` of an OBJECT ProtocolMessage, buffering while not SYNCED.""" + if self._sync_state != ObjectsSyncState.SYNCED: + self._buffered_object_operations.extend(object_messages) # RTO8a + return + self._apply_object_messages(object_messages, ObjectsOperationSource.CHANNEL) # RTO8b + + def _apply_object_messages(self, object_messages: list[ObjectMessage], + source: ObjectsOperationSource) -> None: + """RTO9: applies operations to the pool, recording LOCAL serials in `_applied_on_ack_serials`. + + An operation that raises as it is applied is logged and skipped, and the rest are + applied. + """ + for object_message in object_messages: + try: + self._apply_object_message(object_message, source) + except Exception: + log.exception(f'RealtimeObject._apply_object_messages(): skipping an object message that ' + f'could not be applied; message id={object_message.id}, ' + f'channel={self._channel_name}') + + def _apply_object_message(self, object_message: ObjectMessage, source: ObjectsOperationSource) -> None: + """RTO9a: applies one operation to the pool.""" + operation = object_message.operation + if operation is None: + # RTO9a1 + log.warning(f'RealtimeObject._apply_object_message(): skipping an object message with no ' + f'operation; message id={object_message.id}, channel={self._channel_name}') + return + + serial = object_message.serial + if serial is not None and serial in self._applied_on_ack_serials: + # RTO9a3 + log.debug(f'RealtimeObject._apply_object_message(): skipping an operation already applied ' + f'on ACK; serial={serial}, channel={self._channel_name}') + self._applied_on_ack_serials.discard(serial) + return + + if operation.action not in _SUPPORTED_ACTIONS: + # RTO9a2b + log.warning(f'RealtimeObject._apply_object_message(): skipping an object message with an ' + f'unsupported action; action={operation.action!r}, ' + f'message id={object_message.id}, channel={self._channel_name}') + return + + # RTO9a2a1, RTO9a2a2 + live_object = self._objects_pool.create_zero_value_object_if_not_exists(operation.object_id) + if live_object is None: + return + applied = live_object.apply_operation(object_message, source) # RTO9a2a3 + if source == ObjectsOperationSource.LOCAL and applied: + self._applied_on_ack_serials.add(serial) # RTO9a2a4 + + def _act_on_channel_state(self, state: ChannelState, reason: AblyException | None = None) -> None: + """RTO27: handles the channel entering `state`, for every state but ATTACHED. + + DETACHED and FAILED clear every object's data (RTO27a); every other state keeps it + (RTO27b). DETACHED, SUSPENDED and FAILED fail a `get` or a `_publish_and_apply` + waiting for SYNCED with 92008 (RTO23c1, RTO20e1), with `reason`, else the channel's + `error_reason`, as the cause. DETACHED and FAILED also stop the GC timer, which the + next ATTACHED starts again. + """ + if state in _SYNC_WAIT_FAILURE_STATES: + cause = reason if reason is not None else self._channel_error_reason + self._fail_sync_waiters(_entering(state), cause) + + if state in _DATA_CLEARING_STATES: + self._objects_pool.clear_all_data() # RTO27a1 + self._sync_objects_pool.clear() # RTO27a2 + self._cancel_gc_timer() + + def _set_sync_state(self, state: ObjectsSyncState) -> None: + """RTO17: moves to `state`, emitting the matching `ObjectsEvent` to `on` listeners (RTO17b).""" + if state == self._sync_state: + return + self._sync_state = state + + if state == ObjectsSyncState.SYNCED: + self._resolve_sync_waiters() # RTO23c, RTO20e + + event = _SYNC_STATE_EVENTS.get(state) + if event is None: + return + for registration in self._sync_state_listeners[event].snapshot(): + if not registration.active: + continue + try: + registration.item() # RTO18e + except Exception: + log.exception(f'RealtimeObject._set_sync_state(): a {event.value} listener raised; ' + f'channel={self._channel_name}') + + # Internal API: the sync sequence + + @staticmethod + def _parse_sync_channel_serial(sync_channel_serial: str | None) -> tuple[str | None, str | None]: + """RTO5a1: the `(sequence id, cursor)` of an OBJECT_SYNC's `channelSerial`. + + Both are None when the serial is absent (RTO5a5) or has no `:` separator (RTO5a6). + """ + if sync_channel_serial is None: + return None, None + sync_id, separator, sync_cursor = sync_channel_serial.partition(':') + if not separator: + log.warning(f'RealtimeObject._parse_sync_channel_serial(): handling an OBJECT_SYNC whose ' + f'channelSerial has no ":" as having none; channelSerial={sync_channel_serial!r}') + return None, None + return sync_id, sync_cursor + + def _start_sync_sequence(self, sync_id: str | None) -> None: + """RTO5a2: discards whatever an earlier sync sequence accumulated and starts `sync_id`.""" + self._sync_objects_pool.clear() # RTO5a2a + self._current_sync_id = sync_id + + def _complete_sync(self) -> None: + """RTO5c: applies the objects the sync sequence delivered, then the operations buffered meanwhile. + + An object whose state raises as it is applied is logged and left as the error left it, + so that the sync still completes. + """ + pool = self._objects_pool + received_object_ids: set[str] = set() + updates: list[tuple[LiveObject, LiveObjectUpdate]] = [] + + # RTO5c1 + for object_id, object_message in self._sync_objects_pool.entries.items(): + received_object_ids.add(object_id) + try: + replaced = self._replace_object_state(object_id, object_message) + except Exception: + log.exception(f'RealtimeObject._complete_sync(): skipping an object state that could not be ' + f'applied; object_id={object_id}, message id={object_message.id}, ' + f'channel={self._channel_name}') + continue + if replaced is not None: + updates.append(replaced) + + # RTO5c2, RTO5c2a + for object_id in list(pool): + if object_id not in received_object_ids and object_id != ROOT_OBJECT_ID: + del pool[object_id] + + pool.rebuild_parent_references() # RTO5c10 + + # RTO5c7: emitted once every object is in place, so that listeners see the synced state + for live_object, update in updates: + live_object.notify_updated(update) + + self._apply_object_messages(self._buffered_object_operations, ObjectsOperationSource.CHANNEL) # RTO5c6 + + self._current_sync_id = None # RTO5c3 + self._sync_objects_pool.clear() # RTO5c4 + self._buffered_object_operations = [] # RTO5c5 + self._applied_on_ack_serials.clear() # RTO5c9 + self._set_sync_state(ObjectsSyncState.SYNCED) # RTO5c8 + + def _replace_object_state(self, object_id: str, + object_message: ObjectMessage) -> tuple[LiveObject, LiveObjectUpdate] | None: + """RTO5c1: applies one synced object state to the pool. + + Returns the object and its update when it replaced an existing object's data, whose + update RTO5c7 emits, and None when it created the object. + """ + existing = self._objects_pool.get(object_id) + if existing is not None: + return existing, existing.replace_data(object_message) # RTO5c1a1, RTO5c1a2 + + # RTO5c1b1 + object_state = object_message.object + if object_state.counter is not None: + live_object: LiveObject = InternalLiveCounter(object_id) # RTO5c1b1a + else: + semantics = object_state.map.semantics + live_object = InternalLiveMap( + object_id, semantics if semantics is not None else ObjectsMapSemantics.LWW) # RTO5c1b1b + self._objects_pool[object_id] = live_object + live_object.replace_data(object_message) + return None + + # Internal API: waiting for SYNCED + + async def _wait_for_synced(self, failure_description: str) -> None: + """RTO23c, RTO20e: waits for the sync state to reach SYNCED. + + Raises AblyException 92008, whose message starts with `failure_description`, if the + channel enters DETACHED, SUSPENDED or FAILED first (RTO23c1, RTO20e1). A channel + already DETACHED or FAILED, or released, can never sync without attaching again, so + the wait then fails at once. + """ + if self._released: + raise _sync_wait_failure(failure_description, 'the channel being released', + self._channel_error_reason) + channel = self._channel + if channel is not None and channel.state in _DATA_CLEARING_STATES: + # The channel left ATTACHED before the wait began, as when an ATTACHED or an ACK and a + # channel ERROR are read together + raise _sync_wait_failure(failure_description, _entering(channel.state), channel.error_reason) + + future: asyncio.Future[None] = asyncio.get_running_loop().create_future() + self._sync_waiters[future] = failure_description + try: + await future + finally: + self._sync_waiters.pop(future, None) + + def _resolve_sync_waiters(self) -> None: + waiters, self._sync_waiters = self._sync_waiters, {} + for future in waiters: + if not future.done(): + future.set_result(None) + + def _fail_sync_waiters(self, reason: str, cause: AblyException | None) -> None: + """RTO23c1, RTO20e1: fails every wait for SYNCED with 92008, `reason` saying why.""" + waiters, self._sync_waiters = self._sync_waiters, {} + for future, failure_description in waiters.items(): + if not future.done(): + future.set_exception(_sync_wait_failure(failure_description, reason, cause)) + + @property + def _channel_error_reason(self) -> AblyException | None: + return self._channel.error_reason if self._channel is not None else None + + # Internal API: publishing + + async def _publish(self, object_messages: list[ObjectMessage]) -> PublishResult: + """RTO15: sends `object_messages` in one OBJECT ProtocolMessage and returns the ACK's result. + + Fails as `RealtimeChannel.publish` does when the connection or channel state does not + allow publishing (RTO15b), and with the NACK's error if the publish is rejected (RTO15g). + """ + channel = self._require_channel() + channel._throw_if_unpublishable_state() # RTO15b + + protocol_message = { + 'action': ProtocolMessageAction.OBJECT, # RTO15e1 + 'channel': channel.name, # RTO15e2 + # RTO15c, RTO15e3 + 'state': [object_message.to_dict(self._wire_format) for object_message in object_messages], + } + + log.debug(f'RealtimeObject._publish(): sending {len(object_messages)} object message(s); ' + f'channel={channel.name}') + + # RTO15f, RTO15g, RTO15h + return await channel.ably.connection.connection_manager.send_protocol_message(protocol_message) + + async def _publish_and_apply(self, object_messages: list[ObjectMessage]) -> None: + """RTO20: publishes `object_messages`, then applies them locally with the serials the ACK + assigned, as LOCAL operations, once synced. + + Raises the publish's error (RTO20b), and AblyException 92008 if the channel is or + enters DETACHED or FAILED, or enters SUSPENDED, while waiting for SYNCED (RTO20e1). An + operation that cannot be applied locally is left for its echo to apply (RTO20c, + RTO20d1), and nothing is applied once the channel is DETACHED or FAILED, as its + objects' data has been cleared (RTO27a). + """ + publish_result = await self._publish(object_messages) # RTO20b + + # RTO20c1 + connection_details = self._channel.ably.connection.connection_details + site_code = connection_details.site_code if connection_details is not None else None + if not site_code: + log.error(f'RealtimeObject._publish_and_apply(): the operations will not be applied locally, ' + f'as the connection details carry no siteCode; channel={self._channel_name}') + return + + # RTO20c2 + serials = publish_result.serials if publish_result is not None else None + if serials is None or len(serials) != len(object_messages): + log.error(f'RealtimeObject._publish_and_apply(): the operations will not be applied locally, as the ' + f'ACK carries {len(serials or [])} serial(s) for {len(object_messages)} object message(s); ' + f'channel={self._channel_name}') + return + + # RTO20d + synthetic_messages = [] + for index, (object_message, serial) in enumerate(zip(object_messages, serials)): + if serial is None: + # RTO20d1 + log.debug(f'RealtimeObject._publish_and_apply(): the operation at index {index} will not be ' + f'applied locally, as the ACK assigned it no serial; channel={self._channel_name}') + continue + # RTO20d2, RTO20d3 + synthetic_messages.append(dataclasses.replace(object_message, serial=serial, site_code=site_code)) + + # RTO20d4 + if not synthetic_messages: + return + + if self._sync_state != ObjectsSyncState.SYNCED: + # RTO20e, RTO20e1 + await self._wait_for_synced('The operation could not be applied locally') + + if self._channel.state in _DATA_CLEARING_STATES: + # The channel left ATTACHED after the ACK, and before this resumed + log.debug(f'RealtimeObject._publish_and_apply(): the operations will not be applied locally, as ' + f'the channel is {self._channel.state.value}; channel={self._channel_name}') + return + + self._apply_object_messages(synthetic_messages, ObjectsOperationSource.LOCAL) # RTO20f + + async def _get_server_time_ms(self) -> int: + """RTO16: the current server time, from a persisted offset where one is known (RTO16a). + + The offset is the one this RealtimeObject persisted, else the one the client's `Auth` + persisted for token requests (RSA10k); with neither, the server is asked for its time. + A `RealtimeObject` with no channel reads its own clock. + """ + now_ms = self._clock.now_ms() + if self._channel is None: + return now_ms + + offset_ms = self._server_time_offset_ms + if offset_ms is None: + offset_ms = self._channel.ably.auth.time_offset + if offset_ms is None: + server_time_ms = await self._channel.ably.time() # RTO16 + self._server_time_offset_ms = server_time_ms - self._clock.now_ms() + return int(server_time_ms) + return int(now_ms + offset_ms) # RTO16a + + def _require_channel(self) -> RealtimeChannel: + if self._channel is None: + raise AblyException('Unable to publish object messages from a RealtimeObject with no channel', + 400, 40000) + return self._channel + + # Internal API: preconditions + + def _throw_if_missing_channel_mode(self, mode: ChannelMode) -> None: + """RTO2: raises AblyException 40024 if `mode` is neither granted nor, before ATTACHED, requested. + + The modes the server granted are checked while the channel is ATTACHED and its ATTACHED + carried mode flags (RTO2a); otherwise, the modes requested in the channel options are + (RTO2b). + """ + if self._channel is None: + return + + modes = self._channel.modes if self._channel.state == ChannelState.ATTACHED else None # RTO2a + if not modes: + modes = (self._channel.options or {}).get('modes') or [] # RTO2b + if mode not in modes: + # RTO2a2, RTO2b2 + raise AblyException(f'"{mode.name}" channel mode must be set for this operation', 400, 40024) + + def _check_access_preconditions(self) -> None: + """RTO25: OBJECT_SUBSCRIBE (40024), and 90001 if the channel is DETACHED or FAILED.""" + self._throw_if_missing_channel_mode(ChannelMode.OBJECT_SUBSCRIBE) # RTO25a + self._throw_if_in_channel_state(ChannelState.DETACHED, ChannelState.FAILED) # RTO25b + + def _check_write_preconditions(self) -> None: + """RTO26: OBJECT_PUBLISH (40024), 90001 if the channel is DETACHED, FAILED or SUSPENDED, + and 40000 if `echo_messages` is disabled.""" + self._throw_if_missing_channel_mode(ChannelMode.OBJECT_PUBLISH) # RTO26a + # RTO26b + self._throw_if_in_channel_state(ChannelState.DETACHED, ChannelState.FAILED, ChannelState.SUSPENDED) + if self._channel is not None and not self._channel.ably.options.echo_messages: + # RTO26c + raise AblyException('"echo_messages" client option must be enabled for this operation', 400, 40000) + + def _throw_if_in_channel_state(self, *states: ChannelState) -> None: + if self._channel is not None and self._channel.state in states: + raise AblyException(f'Channel operation failed as channel state is {self._channel.state.value}', + 400, 90001) + + # Internal API: garbage collection + + def _schedule_gc_timer(self) -> None: + """RTO10a: schedules `_on_gc_interval` on the clock, `_gc_interval_ms` from now, if there is a + channel, it has not been released, and no sweep is scheduled already. + + A `RealtimeObject` with no channel schedules nothing. + """ + if self._channel is None or self._released or self._gc_timer is not None: + return + self._gc_timer = self._clock.timer(self._gc_interval_ms, self._on_gc_interval) + + def _cancel_gc_timer(self) -> None: + if self._gc_timer is not None: + self._gc_timer.cancel() + self._gc_timer = None + + def _on_gc_interval(self) -> None: + """RTO10c: releases what has been tombstoned for the grace period, and reschedules itself. + + A sweep that raises is logged, and the next one is scheduled all the same. + """ + self._gc_timer = None + try: + self._objects_pool.collect_garbage(self._gc_grace_period_ms, self._clock.now_ms()) + except Exception: + log.exception(f'RealtimeObject._on_gc_interval(): the GC sweep raised; channel={self._channel_name}') + self._schedule_gc_timer() + + +def _objects_event(event: Any) -> ObjectsEvent: + """`event` as an `ObjectsEvent`, raising AblyException 40003 if it is not one.""" + try: + return ObjectsEvent(event) + except ValueError: + events = ', '.join(repr(member.value) for member in ObjectsEvent) + raise AblyException(f'Objects event should be one of {events}; got {event!r}', 400, 40003) from None + + +def _entering(state: ChannelState) -> str: + """Why a wait for SYNCED fails, when the channel enters `state`.""" + return f'the channel entering the {state.value} state' + + +def _sync_wait_failure(failure_description: str, reason: str, cause: AblyException | None) -> AblyException: + """RTO23c1, RTO20e1: the 92008 error a wait for SYNCED fails with, `reason` saying why.""" + return AblyException(f'{failure_description} due to {reason} whilst waiting for objects sync to complete', + 400, 92008, cause=cause) diff --git a/ably/pubsub/liveobjects/subscription.py b/ably/pubsub/liveobjects/subscription.py new file mode 100644 index 00000000..63dffb1e --- /dev/null +++ b/ably/pubsub/liveobjects/subscription.py @@ -0,0 +1,86 @@ +"""The handles a subscribe call returns, and the registry of listeners behind them.""" + +from __future__ import annotations + +from typing import Callable, Generic, TypeVar + +T = TypeVar('T') + + +class _Handle: + """Calls `deregister` the first time the handle is released, and never again.""" + + def __init__(self, deregister: Callable[[], None]): + self._deregister: Callable[[], None] | None = deregister + + def _release(self) -> None: + deregister, self._deregister = self._deregister, None + if deregister is not None: + deregister() + + +class Subscription(_Handle): + """SUB1: a registration for receiving events from a `subscribe` call.""" + + def unsubscribe(self) -> None: + """SUB2a: deregisters the listener. Calling it again does nothing (SUB2b).""" + self._release() + + +class StatusSubscription(_Handle): + """RTO18f: the registration `RealtimeObject.on` returns.""" + + def off(self) -> None: + """RTO18f1, RTO18f2: deregisters the listener the corresponding `on` call registered.""" + self._release() + + +class Registration(Generic[T]): + """One registration of `item` in a `Registry`, `active` until it is removed.""" + + def __init__(self, item: T): + self.item = item + self.active = True + + +class Registry(Generic[T]): + """The listeners, or other items, that `subscribe`-style calls register, in registration order. + + Registering one item twice makes two registrations, each removed on its own. A dispatch + iterates one `snapshot()` and skips each registration no longer `active` when it is + reached, so that one made during the dispatch is not called by it, and one removed during + it is not called again. + """ + + def __init__(self) -> None: + self._registrations: list[Registration[T]] = [] + + def __bool__(self) -> bool: + return bool(self._registrations) + + def register(self, item: T) -> Callable[[], None]: + """Registers `item`, returning the function that removes this one registration.""" + registration = Registration(item) + self._registrations.append(registration) + return lambda: self._remove(lambda entry: entry is registration) + + def deregister(self, item: T) -> None: + """Removes every registration of `item`.""" + self._remove(lambda entry: entry.item == item) + + def clear(self) -> None: + """Removes every registration.""" + self._remove(lambda entry: True) + + def snapshot(self) -> list[Registration[T]]: + """The registrations as they stand, for a dispatch to iterate.""" + return list(self._registrations) + + def _remove(self, matches: Callable[[Registration[T]], bool]) -> None: + kept = [] + for registration in self._registrations: + if matches(registration): + registration.active = False + else: + kept.append(registration) + self._registrations = kept diff --git a/ably/pubsub/liveobjects/syncobjectspool.py b/ably/pubsub/liveobjects/syncobjectspool.py new file mode 100644 index 00000000..88e54478 --- /dev/null +++ b/ably/pubsub/liveobjects/syncobjectspool.py @@ -0,0 +1,83 @@ +"""The object states accumulated during an objects sync sequence (RTO5f).""" + +from __future__ import annotations + +import logging +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from ably.pubsub.liveobjects.objectmessage import ObjectMessage + +log = logging.getLogger(__name__) + + +class SyncObjectsPool: + """RTO5f: one `ObjectMessage` per object id, received over OBJECT_SYNC and not yet applied. + + A partial map state for an id already held is merged into it (RTO5f2). + """ + + def __init__(self): + self.entries: dict[str, ObjectMessage] = {} + + def apply_object_sync_messages(self, object_messages: list[ObjectMessage]) -> None: + """RTO5f: stores each message with an `object`, merging partial map states (RTO5f2a) and + skipping messages with neither `map` nor `counter` (RTO5f3) and partial counter states + (RTO5f2b).""" + for object_message in object_messages: + object_state = object_message.object + if object_state is None: + # RTO5d: only this message is skipped, so the rest of the sequence still applies + log.warning(f'SyncObjectsPool.apply_object_sync_messages(): skipping an OBJECT_SYNC message ' + f'with no object; message id={object_message.id}') + continue + + if object_state.map is None and object_state.counter is None: + # RTO5f3 + log.warning(f'SyncObjectsPool.apply_object_sync_messages(): skipping an OBJECT_SYNC message ' + f'with an unsupported object type; object_id={object_state.object_id}, ' + f'message id={object_message.id}') + continue + + existing = self.entries.get(object_state.object_id) + if existing is None: + self.entries[object_state.object_id] = object_message # RTO5f1 + continue + + # RTO5f2: a partial state of an object split across several OBJECT_SYNC messages + if object_state.map is not None: + self._merge_partial_map_state(existing, object_message) # RTO5f2a + else: + # RTO5f2b + log.error(f'SyncObjectsPool.apply_object_sync_messages(): skipping an unexpected partial ' + f'state for a counter; object_id={object_state.object_id}, ' + f'message id={object_message.id}') + + def clear(self) -> None: + """RTO5a2a, RTO5c4, RTO27a2: discards every stored message.""" + self.entries.clear() + + def _merge_partial_map_state(self, existing: ObjectMessage, object_message: ObjectMessage) -> None: + """RTO5f2a: merges the partial map state in `object_message` into the stored `existing`.""" + object_state = object_message.object + if object_state.tombstone: + self.entries[object_state.object_id] = object_message # RTO5f2a1 + return + + existing_map = existing.object.map + if existing_map is None: + log.error(f'SyncObjectsPool.apply_object_sync_messages(): skipping a partial map state for an ' + f'object first received as a counter; object_id={object_state.object_id}, ' + f'message id={object_message.id}') + return + + # RTO5f2a2: no two partial states of one map carry the same key + if existing_map.entries is None: + existing_map.entries = {} + existing_map.entries.update(object_state.map.entries or {}) + + def __len__(self) -> int: + return len(self.entries) + + def __contains__(self, object_id: object) -> bool: + return object_id in self.entries diff --git a/ably/pubsub/liveobjects/valuetypes.py b/ably/pubsub/liveobjects/valuetypes.py new file mode 100644 index 00000000..176f44f5 --- /dev/null +++ b/ably/pubsub/liveobjects/valuetypes.py @@ -0,0 +1,289 @@ +"""The creation blueprints `LiveCounter` and `LiveMap` (RTLCV*, RTLMV*), and the value unions. + +A blueprint is an inert, immutable description of an object to create. It becomes +ObjectMessages only when a mutation evaluates it (RTLCV4, RTLMV4), so `create` performs +no validation (RTLCV3c, RTLMV3c). + +Evaluation is pure: the caller fetches the server time (RTO16) and passes it in, and +each evaluation generates a fresh nonce, so a blueprint used twice creates two objects. +""" + +from __future__ import annotations + +import json +import math +from dataclasses import dataclass +from typing import Any, Dict, List, TypeVar, Union + +from ably.pubsub.liveobjects.objectid import generate_nonce, generate_object_id +from ably.pubsub.liveobjects.objectmessage import ( + WIRE_FORMAT_JSON, + CounterCreate, + CounterCreateWithObjectId, + MapCreate, + MapCreateWithObjectId, + ObjectData, + ObjectMessage, + ObjectOperation, + ObjectOperationAction, + ObjectsMapEntry, + ObjectsMapSemantics, +) +from ably.pubsub.util.exceptions import AblyException + +# The primitive values a map entry can hold: String, Number, Boolean, Binary, JsonArray +# and JsonObject (RTLM20a3) +Primitive = Union[str, float, bool, bytes, list, dict] + +T = TypeVar('T', bound=Primitive) + +# A JSON value and a JSON object, as `compact_json` returns them (RTPO14, RTINS11) +JsonValue = Union[str, float, bool, None, List['JsonValue'], Dict[str, 'JsonValue']] +JsonObject = Dict[str, JsonValue] + + +@dataclass(frozen=True) +class LiveCounter: + """RTLCV1: the intent to create a counter with an initial count.""" + + _count: Any = 0 # RTLCV2a + + @staticmethod + def create(initial_count: float = 0) -> LiveCounter: + """RTLCV3: a blueprint for a counter starting at `initial_count`.""" + return LiveCounter(initial_count) + + +@dataclass(frozen=True) +class LiveMap: + """RTLMV1: the intent to create a map with initial entries.""" + + _entries: Any = None # RTLMV2a + + @staticmethod + def create(entries: dict[str, LiveMapValue] | None = None) -> LiveMap: + """RTLMV3: a blueprint for a map holding `entries`. + + A dict is copied, along with each JSON object, JSON array and binary value in it, so + the blueprint does not change when the caller's values do (RTLMV3d); anything else is + kept for evaluation to reject (RTLMV3c). + """ + if not isinstance(entries, dict): + return LiveMap(entries) + return LiveMap({key: _snapshot(value) for key, value in entries.items()}) + + +# The values `set` accepts: a primitive, or a blueprint for a new object (RTTS11) +LiveMapValue = Union[Primitive, LiveMap, LiveCounter] + + +def validate_key(key: Any) -> None: + """RTLMV4b: raises AblyException 40003 unless `key` is a string. + + A map's `set` and `remove` validate their key the same way (RTLM20e1, RTLM21e1). + """ + if not isinstance(key, str): + raise AblyException('Map key should be string', 400, 40003) + + +def validate_value(value: Any) -> None: + """RTLM20e1: raises the AblyException a map's `set` raises for `value`, without evaluating it. + + That is 40013 unless `value` is a primitive or a blueprint (RTLMV4c), and for a blueprint, + whatever evaluating it raises (RTLCV4a, RTLMV4a-RTLMV4c), so that a write can reject one + before fetching the server time its object ids are generated from (RTO16). + """ + if isinstance(value, LiveCounter): + _counter_create(value) + elif isinstance(value, LiveMap): + entries, _ = _map_entries_data(value) + for entry in entries.values(): + if isinstance(entry, (LiveCounter, LiveMap)): + validate_value(entry) + else: + primitive_to_object_data(value) + + +def validate_amount(amount: Any) -> float: + """RTLC12e1: `amount` as a float, raising AblyException 40003 unless it is a finite number. + + A bool is not a number, and None is not an omitted amount. A counter's `increment` and + `decrement` validate their amount this way (RTLC12e1, RTLC13c). + """ + number = _finite_number(amount) + if number is None: + raise AblyException('Counter value increment should be a valid number', 400, 40003) + return number + + +def primitive_to_object_data(value: Any) -> ObjectData: + """RTLMV4d3-RTLMV4d7: the `ObjectData` holding the primitive `value`. + + It holds the value as it decodes on receipt: a number as a float (OD4c3), binary as + `bytes`, and a JSON object or array as a copy parsed back from its JSON encoding. + Raises AblyException 40013 for any other value (RTLMV4c, OD4a), which includes a + number that is not finite, a dict or list that is not JSON-encodable, a blueprint, + and a live object or a `PathObject` or `Instance` wrapping one (RTLMV4c1). + """ + if isinstance(value, bool): + return ObjectData(boolean=value) # RTLMV4d6 + if isinstance(value, (int, float)): + number = _finite_number(value) + if number is not None: + return ObjectData(number=number) # RTLMV4d5 + elif isinstance(value, str): + return ObjectData(string=value) # RTLMV4d4 + elif isinstance(value, (bytes, bytearray)): + return ObjectData(bytes=bytes(value)) # RTLMV4d7 + elif isinstance(value, (dict, list)): + try: + encoded = json.dumps(value, allow_nan=False) + except (TypeError, ValueError, RecursionError): + pass + else: + return ObjectData(json=json.loads(encoded)) # RTLMV4d3 + raise AblyException('Map value data type is unsupported', 400, 40013) + + +def evaluate_live_counter(value: LiveCounter, timestamp_ms: int) -> ObjectMessage: + """RTLCV4: the COUNTER_CREATE `ObjectMessage` a `LiveCounter` evaluates to. + + `timestamp_ms` is the server time the object id is generated with (RTLCV4e). The + operation carries `counter_create_with_object_id`, whose `derived_from` is the + `CounterCreate` it was built from (RTLCV4g5). Raises AblyException 40003 for a count + that is not a finite number (RTLCV4a). + """ + counter_create = _counter_create(value) # RTLCV4a, RTLCV4b + initial_value = _initial_value(counter_create) # RTLCV4c + nonce = generate_nonce() # RTLCV4d + object_id = generate_object_id('counter', initial_value, nonce, timestamp_ms) # RTLCV4f + return ObjectMessage(operation=ObjectOperation( + action=ObjectOperationAction.COUNTER_CREATE, # RTLCV4g1 + object_id=object_id, # RTLCV4g2 + counter_create_with_object_id=CounterCreateWithObjectId( + initial_value=initial_value, # RTLCV4g4 + nonce=nonce, # RTLCV4g3 + derived_from=counter_create, # RTLCV4g5 + ), + )) + + +def evaluate_live_map(value: LiveMap, timestamp_ms: int) -> list[ObjectMessage]: + """RTLMV4: the ObjectMessages a `LiveMap` evaluates to, nested creates first, depth-first, + and this map's MAP_CREATE last (RTLMV4k). + + The MAP_CREATE carries `map_create_with_object_id`, whose `derived_from` is the + `MapCreate` it was built from (RTLMV4j5). Raises AblyException 40003 for entries that + are not a dict or a key that is not a string (RTLMV4a, RTLMV4b), and 40013 for a + value of an unsupported type (RTLMV4c). + """ + # Every value at this level is validated (RTLMV4c) before any blueprint is evaluated + entries, data = _map_entries_data(value) + + messages: list[ObjectMessage] = [] + for key, entry in entries.items(): + if isinstance(entry, LiveCounter): + counter_create_message = evaluate_live_counter(entry, timestamp_ms) # RTLMV4d1 + messages.append(counter_create_message) + data[key] = ObjectData(object_id=counter_create_message.operation.object_id) + elif isinstance(entry, LiveMap): + nested_messages = evaluate_live_map(entry, timestamp_ms) # RTLMV4d2 + messages.extend(nested_messages) + data[key] = ObjectData(object_id=nested_messages[-1].operation.object_id) + + map_create = MapCreate( # RTLMV4e + semantics=ObjectsMapSemantics.LWW, + entries={key: ObjectsMapEntry(data=entry_data) for key, entry_data in data.items()}, + ) + initial_value = _initial_value(map_create) # RTLMV4f + nonce = generate_nonce() # RTLMV4g + object_id = generate_object_id('map', initial_value, nonce, timestamp_ms) # RTLMV4i + messages.append(ObjectMessage(operation=ObjectOperation( + action=ObjectOperationAction.MAP_CREATE, # RTLMV4j1 + object_id=object_id, # RTLMV4j2 + map_create_with_object_id=MapCreateWithObjectId( + initial_value=initial_value, # RTLMV4j4 + nonce=nonce, # RTLMV4j3 + derived_from=map_create, # RTLMV4j5 + ), + ))) + return messages # RTLMV4k + + +def evaluate(value: LiveCounter | LiveMap, timestamp_ms: int) -> list[ObjectMessage]: + """The ObjectMessages a blueprint evaluates to, the create for `value` itself last.""" + if isinstance(value, LiveCounter): + return [evaluate_live_counter(value, timestamp_ms)] + if isinstance(value, LiveMap): + return evaluate_live_map(value, timestamp_ms) + raise TypeError(f'Expected a LiveCounter or LiveMap, got {type(value).__name__}') + + +def _counter_create(value: LiveCounter) -> CounterCreate: + """RTLCV4a, RTLCV4b: the `CounterCreate` a `LiveCounter` describes, raising AblyException 40003 + for a count that is not a finite number.""" + count = _finite_number(value._count) + if count is None: + raise AblyException('Counter value should be a valid number', 400, 40003) # RTLCV4a + return CounterCreate(count=count) # RTLCV4b + + +def _map_entries_data(value: LiveMap) -> tuple[dict, dict[str, ObjectData | None]]: + """RTLMV4a-RTLMV4c: a `LiveMap`'s entries, and the `ObjectData` of each, None for a blueprint. + + Raises AblyException 40003 for entries that are not a dict or a key that is not a string, + and 40013 for a value of an unsupported type at this level; a blueprint among the entries + is not looked into. + """ + # `LiveMap.create()` and `LiveMap.create(None)` both leave the entries unset + entries = {} if value._entries is None else value._entries + if not isinstance(entries, dict): + raise AblyException('Map entries should be a dict', 400, 40003) # RTLMV4a + for key in entries: + validate_key(key) # RTLMV4b + # RTLMV4c + data: dict[str, ObjectData | None] = { + key: None if isinstance(entry, (LiveCounter, LiveMap)) else primitive_to_object_data(entry) + for key, entry in entries.items() + } + return entries, data + + +def _snapshot(value: Any) -> Any: + """A copy of a map entry's value that later changes to `value` do not reach (RTLMV3d). + + A JSON object or array is copied through its JSON encoding, which evaluation applies to + it anyway (RTLMV4d3), and binary is copied to `bytes`. A value that does not encode is + kept as it is, as no validation is done at creation (RTLMV3c); evaluation rejects it. + """ + if isinstance(value, bytearray): + return bytes(value) + if isinstance(value, (dict, list)): + try: + return json.loads(json.dumps(value, allow_nan=False)) + except (TypeError, ValueError, RecursionError): + return value + return value + + +def _finite_number(value: Any) -> float | None: + """`value` as a float if it is a finite number, else None. A boolean is not a number.""" + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + try: + number = float(value) + except OverflowError: + return None + return number if math.isfinite(number) else None + + +def _initial_value(create: CounterCreate | MapCreate) -> str: + """RTLCV4c, RTLMV4f: the JSON string of `create` in its JSON-wire encoding (OD4d), whatever + protocol the connection uses. + + The object id is the hash of this exact string (RTO14b1), and it is sent unchanged as + the `initialValue`. Keys keep the order `to_dict` builds them in, with no whitespace + between tokens, non-ASCII characters escaped, and numbers in Python's shortest + round-trip form. + """ + return json.dumps(create.to_dict(WIRE_FORMAT_JSON), separators=(',', ':'), allow_nan=False) diff --git a/ably/pubsub/prototypes.py b/ably/pubsub/prototypes.py new file mode 100644 index 00000000..4fc2613e --- /dev/null +++ b/ably/pubsub/prototypes.py @@ -0,0 +1,131 @@ +"""The shapes of the Pub/Sub clients, as callers see them. + +The client classes themselves are internal: they are reachable only through the +factory functions in :mod:`ably.pubsub.server` and :mod:`ably.pubsub.server.sync`, +so that the package a client comes from names the side the application runs on. +These protocols are what those factories are declared to return, and what +application code should annotate against. + +The clients name their prototype as a base class rather than satisfying it +structurally, so the relationship is stated where the client is defined and a +type checker verifies it. One consequence to know about: an unimplemented +member resolves to the stub below and returns None instead of raising, so +``test/unit/pubsub_server_test.py`` checks that each client defines every +member itself, and with the signature declared here. + +They are named "prototypes" rather than "protocols" because this codebase +already uses "protocol" for the Ably wire protocol. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Protocol + +if TYPE_CHECKING: + from ably.pubsub.http.auth import Auth + from ably.pubsub.http.channel import Channels as HttpChannels + from ably.pubsub.http.push import Push + from ably.pubsub.realtime.channel import Channels as RealtimeChannels + from ably.pubsub.realtime.connection import Connection + from ably.pubsub.request.paginatedresult import HttpPaginatedResponse, PaginatedResult + from ably.pubsub.types.batch import BatchPublishSpec, BatchResult + from ably.pubsub.types.options import Options + + +class PubSubHttpClient(Protocol): + """A Pub/Sub client that operates entirely over HTTP. + + Build one with ``create_http_client``, from :mod:`ably.pubsub.server` for the + asynchronous flavour or :mod:`ably.pubsub.server.sync` for the synchronous one. + """ + + @property + def auth(self) -> Auth: + """The authentication object for this client.""" + ... + + @property + def channels(self) -> HttpChannels: + """The channels container object.""" + ... + + @property + def client_id(self) -> str | None: + """The client ID this client authenticates as, if any.""" + ... + + @property + def options(self) -> Options: + """The client options this client was built with.""" + ... + + @property + def push(self) -> Push: + """The push notification admin object.""" + ... + + async def stats(self, direction: str | None = None, start=None, end=None, params: dict | None = None, + limit: int | None = None, paginated=None, unit=None, timeout=None) -> PaginatedResult: + """Return the stats for this application.""" + ... + + async def time(self, timeout: float | None = None) -> float: + """Return the current server time in ms since the unix epoch.""" + ... + + async def batch_publish(self, specs: BatchPublishSpec | dict | list[BatchPublishSpec | dict] + ) -> BatchResult | list[BatchResult]: + """Publish messages to one or more channels in a single request.""" + ... + + async def batch_presence(self, channels: list[str]) -> BatchResult: + """Return the presence members of several channels in a single request.""" + ... + + async def request(self, method: str, path: str, version: str, params: dict | None = None, + body=None, headers=None) -> HttpPaginatedResponse: + """Make an arbitrary request against the Ably REST API.""" + ... + + async def close(self) -> None: + """Release the resources this client holds.""" + ... + + async def __aenter__(self) -> PubSubHttpClient: + ... + + async def __aexit__(self, *excinfo) -> None: + ... + + +class PubSubRealtimeClient(PubSubHttpClient, Protocol): + """A Pub/Sub client with a persistent realtime connection. + + Everything :class:`PubSubHttpClient` does, plus subscribing to channels and + entering presence. Build one with + :func:`ably.pubsub.server.create_realtime_client`. + """ + + @property + def channels(self) -> RealtimeChannels: + """The realtime channels container object.""" + ... + + @property + def connection(self) -> Connection: + """The realtime connection object.""" + ... + + def connect(self) -> None: + """Establish the realtime connection. + + Unnecessary unless the client was built with ``auto_connect=False``. + """ + ... + + async def close(self) -> None: + """Close the realtime connection and release the client's resources.""" + ... + + async def __aenter__(self) -> PubSubRealtimeClient: + ... diff --git a/ably/realtime/__init__.py b/ably/pubsub/realtime/__init__.py similarity index 100% rename from ably/realtime/__init__.py rename to ably/pubsub/realtime/__init__.py diff --git a/ably/realtime/annotations.py b/ably/pubsub/realtime/annotations.py similarity index 92% rename from ably/realtime/annotations.py rename to ably/pubsub/realtime/annotations.py index 57400d45..cf83c609 100644 --- a/ably/realtime/annotations.py +++ b/ably/pubsub/realtime/annotations.py @@ -3,17 +3,17 @@ import logging from typing import TYPE_CHECKING -from ably.rest.annotations import RestAnnotations, construct_validate_annotation -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.channelmode import ChannelMode -from ably.types.channelstate import ChannelState -from ably.util.eventemitter import EventEmitter -from ably.util.helper import is_callable_or_coroutine +from ably.pubsub.http.annotations import HttpAnnotations, construct_validate_annotation +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.eventemitter import EventEmitter +from ably.pubsub.util.helper import is_callable_or_coroutine if TYPE_CHECKING: - from ably.realtime.channel import RealtimeChannel - from ably.realtime.connectionmanager import ConnectionManager + from ably.pubsub.realtime.channel import RealtimeChannel + from ably.pubsub.realtime.connectionmanager import ConnectionManager log = logging.getLogger(__name__) @@ -37,7 +37,7 @@ def __init__(self, channel: RealtimeChannel, connection_manager: ConnectionManag self.__channel = channel self.__connection_manager = connection_manager self.__subscriptions = EventEmitter() - self.__rest_annotations = RestAnnotations(channel) + self.__http_annotations = HttpAnnotations(channel) async def __send_annotation(self, annotation: Annotation, params: dict | None = None): """ @@ -264,4 +264,4 @@ async def get(self, msg_or_serial, params: dict | None = None): AblyException: If the request fails or serial is invalid """ # Delegate to REST implementation - return await self.__rest_annotations.get(msg_or_serial, params) + return await self.__http_annotations.get(msg_or_serial, params) diff --git a/ably/realtime/channel.py b/ably/pubsub/realtime/channel.py similarity index 86% rename from ably/realtime/channel.py rename to ably/pubsub/realtime/channel.py index fde9bb06..e484f656 100644 --- a/ably/realtime/channel.py +++ b/ably/pubsub/realtime/channel.py @@ -4,28 +4,30 @@ import logging from typing import TYPE_CHECKING -from ably.realtime.annotations import RealtimeAnnotations -from ably.realtime.connection import ConnectionState -from ably.realtime.presence import RealtimePresence -from ably.rest.channel import Channel -from ably.rest.channel import Channels as RestChannels -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.annotation import Annotation -from ably.types.channelmode import ChannelMode, decode_channel_mode, encode_channel_mode -from ably.types.channeloptions import ChannelOptions -from ably.types.channelstate import ChannelState, ChannelStateChange -from ably.types.flags import Flag, has_flag -from ably.types.message import Message, MessageAction, MessageVersion -from ably.types.mixins import DecodingContext -from ably.types.operations import MessageOperation, PublishResult, UpdateDeleteResult -from ably.types.presence import PresenceMessage -from ably.util.clock import select_clock -from ably.util.eventemitter import EventEmitter -from ably.util.exceptions import AblyException, IncompatibleClientIdException -from ably.util.helper import Timer, is_callable_or_coroutine, validate_message_size +from ably.pubsub.http.channel import Channel +from ably.pubsub.http.channel import Channels as HttpChannels +from ably.pubsub.liveobjects.objectmessage import ObjectMessage +from ably.pubsub.liveobjects.realtimeobject import RealtimeObject +from ably.pubsub.realtime.annotations import RealtimeAnnotations +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.realtime.presence import RealtimePresence +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.annotation import Annotation +from ably.pubsub.types.channelmode import ChannelMode, decode_channel_mode, encode_channel_mode +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.channelstate import ChannelState, ChannelStateChange +from ably.pubsub.types.flags import Flag, has_flag +from ably.pubsub.types.message import Message, MessageAction, MessageVersion +from ably.pubsub.types.mixins import DecodingContext +from ably.pubsub.types.operations import MessageOperation, PublishResult, UpdateDeleteResult +from ably.pubsub.types.presence import PresenceMessage +from ably.pubsub.util.clock import select_clock +from ably.pubsub.util.eventemitter import EventEmitter +from ably.pubsub.util.exceptions import AblyException, IncompatibleClientIdException +from ably.pubsub.util.helper import Timer, is_callable_or_coroutine, validate_message_size if TYPE_CHECKING: - from ably.realtime.realtime import AblyRealtime + from ably.pubsub.realtime.realtime import DefaultPubSubRealtimeClient log = logging.getLogger(__name__) @@ -55,7 +57,8 @@ class RealtimeChannel(EventEmitter, Channel): Unsubscribe to messages from a channel """ - def __init__(self, realtime: AblyRealtime, name: str, channel_options: ChannelOptions | None = None): + def __init__(self, realtime: DefaultPubSubRealtimeClient, name: str, + channel_options: ChannelOptions | None = None): EventEmitter.__init__(self) self.__name = name self.__realtime = realtime @@ -91,6 +94,9 @@ def __init__(self, realtime: AblyRealtime, name: str, channel_options: ChannelOp # Initialize realtime annotations for this channel (override REST annotations) self._Channel__annotations = RealtimeAnnotations(self, realtime.connection.connection_manager) + # RTL27: the LiveObjects entry point for this channel + self.__object = RealtimeObject(self) + async def set_options(self, channel_options: ChannelOptions) -> None: """Set channel options""" should_reattach = self.should_reattach_to_set_options(channel_options) @@ -120,12 +126,21 @@ async def attach(self) -> None: AblyException If unable to attach channel """ + await self._attach() + + async def _attach(self) -> ChannelStateChange | None: + """The attach procedure `attach` runs, returning the state change it ended with, or None if + the channel was already ATTACHED. + Raises where `attach` does. An attach that ends in a state other than ATTACHED, + SUSPENDED or FAILED, such as DETACHED when the connection closes first, returns that + state change. + """ log.info(f'RealtimeChannel.attach() called, channel = {self.name}') # RTL4a - if channel is attached do nothing if self.state == ChannelState.ATTACHED: - return + return None self.__error_reason = None @@ -149,6 +164,7 @@ async def attach(self) -> None: if state_change.current in (ChannelState.SUSPENDED, ChannelState.FAILED): raise state_change.reason + return state_change def _attach_impl(self): log.debug("RealtimeChannel.attach_impl(): sending ATTACH protocol message") @@ -169,6 +185,33 @@ def _attach_impl(self): self._send_message(attach_msg) + # RTL33 + async def _ensure_active(self) -> None: + """The ensure-active-channel procedure: attaches the channel unless it is ATTACHED or SUSPENDED. + + Raises + ------ + AblyException + 90001 if the channel is FAILED (RTL33c), or the error the implicit attach failed + with (RTL33b1): the reason for the state it ended in, else 90001 + """ + # RTL33a + if self.state in (ChannelState.ATTACHED, ChannelState.SUSPENDED): + return + + # RTL33c + if self.state == ChannelState.FAILED: + raise AblyException(f"Channel operation failed as channel state is {self.state.value}", 400, 90001) + + # RTL33b + state_change = await self._attach() + if state_change is not None and state_change.current != ChannelState.ATTACHED: + # RTL33b1, RTL4d: the attach ended without the channel attaching, as when the connection + # closes or `detach` is called first + raise state_change.reason or AblyException( + f'Unable to attach channel; channel state = {state_change.current.value}, ' + f'connection state = {self.__realtime.connection.state.value}', 400, 90001) + # RTL5 async def detach(self) -> None: """Detach from channel @@ -409,7 +452,7 @@ async def publish(self, *args, **kwargs) -> PublishResult: 400, 40012) - # Encode messages (RTL6a: same encoding as RestChannel#publish) + # Encode messages (RTL6a: same encoding as HttpChannel#publish) encoded_messages = [] for m in messages: # Encode the message with encryption if needed @@ -705,6 +748,7 @@ def _on_message(self, proto_msg: dict) -> None: exception = None resumed = False has_presence = False + has_objects = False self.__attach_serial = channel_serial self.__channel_serial = channel_serial @@ -717,6 +761,8 @@ def _on_message(self, proto_msg: dict) -> None: resumed = has_flag(flags, Flag.RESUMED) # RTP1: Check for HAS_PRESENCE flag has_presence = has_flag(flags, Flag.HAS_PRESENCE) + # RTO4: whether an OBJECT_SYNC follows + has_objects = has_flag(flags, Flag.HAS_OBJECTS) # Store channel attach flags self.__modes = decode_channel_mode(flags) @@ -725,8 +771,11 @@ def _on_message(self, proto_msg: dict) -> None: if not resumed: state_change = ChannelStateChange(self.state, ChannelState.ATTACHED, resumed, exception) self._emit("update", state_change) + # RTO4: an ATTACHED restarts the objects sync even while attached, a resumed one included + self.__object._on_attached(has_objects) elif self.state == ChannelState.ATTACHING: self._notify_state(ChannelState.ATTACHED, resumed=resumed, has_presence=has_presence) + self.__object._on_attached(has_objects) # RTO4 else: log.warn("RealtimeChannel._on_message(): ATTACHED received while not attaching") elif action == ProtocolMessageAction.DETACHED: @@ -773,6 +822,18 @@ def _on_message(self, proto_msg: dict) -> None: self.__channel_serial = channel_serial except Exception as e: log.error(f"Annotation processing error {e}. Skip annotations {annotation_data}") + elif action == ProtocolMessageAction.OBJECT: + # RTL15b + if channel_serial: + self.__channel_serial = channel_serial + # An object message that fails to decode is logged and left out, and the rest are handled + object_messages = ObjectMessage.from_protocol_message(proto_msg, self.__object._wire_format) + self.__object._handle_object_messages(object_messages) # RTO8 + elif action == ProtocolMessageAction.OBJECT_SYNC: + # RTO5: the channelSerial of an OBJECT_SYNC carries the sync sequence and cursor (RTO5a1), + # which are handled whichever of its object messages fail to decode + object_messages = ObjectMessage.from_protocol_message(proto_msg, self.__object._wire_format) + self.__object._handle_object_sync_messages(object_messages, channel_serial) elif action == ProtocolMessageAction.ERROR: error = AblyException.from_dict(proto_msg.get('error')) self._notify_state(ChannelState.FAILED, reason=error) @@ -824,6 +885,10 @@ def _notify_state(self, state: ChannelState, reason: AblyException | None = None # RTP5: Notify presence of channel state change self.__presence.act_on_channel_state(state, has_presence=has_presence, error=reason) + # RTO27: the objects act on every state but ATTACHED, whose ATTACHED message RTO4 handles + if state != ChannelState.ATTACHED: + self.__object._act_on_channel_state(state, reason) + def _send_message(self, msg: dict) -> None: asyncio.create_task(self.__realtime.connection.connection_manager.send_protocol_message(msg)) @@ -924,6 +989,12 @@ def presence(self): def annotations(self) -> RealtimeAnnotations: return self._Channel__annotations + # RTL27 + @property + def object(self) -> RealtimeObject: + """The LiveObjects on this channel""" + return self.__object + @property def modes(self): """Get the list of channel modes""" @@ -962,7 +1033,7 @@ def _encode_flags(self) -> int | None: return flags -class Channels(RestChannels): +class Channels(HttpChannels): """Creates and destroys RealtimeChannel objects. Methods @@ -1014,17 +1085,36 @@ def release(self, name: str) -> None: """Releases a RealtimeChannel object, deleting it, and enabling it to be garbage collected It also removes any listeners associated with the channel. - To release a channel, the channel state must be INITIALIZED, DETACHED, or FAILED. - + A realtime channel can only be released when it is in the INITIALIZED, DETACHED, or FAILED state. Parameters ---------- name: str Channel name + + Raises + ------ + AblyException + With code 90011 if the channel is in any other state. Await channel.detach() before + calling release(). """ + # RTS4c if name not in self.__all: return + channel = self.__all[name] + # RTS4e + if channel.state not in (ChannelState.INITIALIZED, ChannelState.DETACHED, ChannelState.FAILED): + raise AblyException( + 'Can only release a channel in a state where there is no possibility of further updates ' + 'from the server being received (initialized, detached, or failed). The current state is ' + f'{channel.state.value}', + 400, + 90011, + ) + # RTS4d del self.__all[name] + # The released channel receives nothing more, so its objects stop their GC timer and their waits + channel.object._release() def _on_channel_message(self, msg: dict) -> None: channel_name = msg.get('channel') @@ -1066,8 +1156,10 @@ def _propagate_connection_interruption(self, state: ConnectionState, reason: Abl channel._notify_state(connection_to_channel_state[state], reason) def _on_connected(self) -> None: + connection_details = self.__ably.connection.connection_details for channel_name in self.__all: channel = self.__all[channel_name] + channel.object._on_connected(connection_details) # RTO10b2 if channel.state == ChannelState.ATTACHING or channel.state == ChannelState.DETACHING: channel._check_pending_state() elif channel.state == ChannelState.SUSPENDED: diff --git a/ably/realtime/connection.py b/ably/pubsub/realtime/connection.py similarity index 89% rename from ably/realtime/connection.py rename to ably/pubsub/realtime/connection.py index 907f56a5..fb7854df 100644 --- a/ably/realtime/connection.py +++ b/ably/pubsub/realtime/connection.py @@ -5,14 +5,14 @@ import logging from typing import TYPE_CHECKING -from ably.realtime.connectionmanager import ConnectionManager -from ably.types.connectiondetails import ConnectionDetails -from ably.types.connectionstate import ConnectionEvent, ConnectionState, ConnectionStateChange -from ably.util.eventemitter import EventEmitter -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connectionmanager import ConnectionManager +from ably.pubsub.types.connectiondetails import ConnectionDetails +from ably.pubsub.types.connectionstate import ConnectionEvent, ConnectionState, ConnectionStateChange +from ably.pubsub.util.eventemitter import EventEmitter +from ably.pubsub.util.exceptions import AblyException if TYPE_CHECKING: - from ably.realtime.realtime import AblyRealtime + from ably.pubsub.realtime.realtime import DefaultPubSubRealtimeClient log = logging.getLogger(__name__) @@ -40,7 +40,7 @@ class Connection(EventEmitter): # RTN4 Pings a realtime connection """ - def __init__(self, realtime: AblyRealtime): + def __init__(self, realtime: DefaultPubSubRealtimeClient): self.__realtime = realtime self.__error_reason: AblyException | None = None self.__state = ConnectionState.CONNECTING if realtime.options.auto_connect else ConnectionState.INITIALIZED diff --git a/ably/realtime/connectionmanager.py b/ably/pubsub/realtime/connectionmanager.py similarity index 97% rename from ably/realtime/connectionmanager.py rename to ably/pubsub/realtime/connectionmanager.py index 9f920b07..407692d2 100644 --- a/ably/realtime/connectionmanager.py +++ b/ably/pubsub/realtime/connectionmanager.py @@ -8,20 +8,20 @@ import httpx -from ably.transport.defaults import Defaults -from ably.transport.websockettransport import ProtocolMessageAction, WebSocketTransport -from ably.types.connectiondetails import ConnectionDetails -from ably.types.connectionerrors import ConnectionErrors -from ably.types.connectionstate import ConnectionEvent, ConnectionState, ConnectionStateChange -from ably.types.operations import PublishResult -from ably.types.tokendetails import TokenDetails -from ably.util.clock import Clock, select_clock -from ably.util.eventemitter import EventEmitter -from ably.util.exceptions import AblyException, IncompatibleClientIdException -from ably.util.helper import Timer, get_random_id, is_token_error +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.transport.websockettransport import ProtocolMessageAction, WebSocketTransport +from ably.pubsub.types.connectiondetails import ConnectionDetails +from ably.pubsub.types.connectionerrors import ConnectionErrors +from ably.pubsub.types.connectionstate import ConnectionEvent, ConnectionState, ConnectionStateChange +from ably.pubsub.types.operations import PublishResult +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.clock import Clock, select_clock +from ably.pubsub.util.eventemitter import EventEmitter +from ably.pubsub.util.exceptions import AblyException, IncompatibleClientIdException +from ably.pubsub.util.helper import Timer, get_random_id, is_token_error if TYPE_CHECKING: - from ably.realtime.realtime import AblyRealtime + from ably.pubsub.realtime.realtime import DefaultPubSubRealtimeClient log = logging.getLogger(__name__) @@ -139,7 +139,7 @@ def response_time_ms(self) -> float: class ConnectionManager(EventEmitter): - def __init__(self, realtime: AblyRealtime, initial_state): + def __init__(self, realtime: DefaultPubSubRealtimeClient, initial_state): self.options = realtime.options self.clock = select_clock(self.options) self.__ably = realtime @@ -211,6 +211,8 @@ async def __get_transport_params(self) -> dict: # RTN2a: Set format to msgpack if use_binary_protocol is enabled if self.options.use_binary_protocol: params["format"] = "msgpack" + # RTN2h, RTC1a: whether the server echoes this connection's own messages back to it + params["echo"] = "true" if self.options.echo_messages else "false" # Add any custom transport params from options params.update(self.options.transport_params) diff --git a/ably/realtime/presence.py b/ably/pubsub/realtime/presence.py similarity index 98% rename from ably/realtime/presence.py rename to ably/pubsub/realtime/presence.py index 79d73070..60b3b211 100644 --- a/ably/realtime/presence.py +++ b/ably/pubsub/realtime/presence.py @@ -12,15 +12,15 @@ from datetime import datetime, timezone from typing import TYPE_CHECKING, Any -from ably.realtime.connection import ConnectionState -from ably.realtime.presencemap import PresenceMap -from ably.types.channelstate import ChannelState, ChannelStateChange -from ably.types.presence import PresenceAction, PresenceMessage -from ably.util.eventemitter import EventEmitter -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.realtime.presencemap import PresenceMap +from ably.pubsub.types.channelstate import ChannelState, ChannelStateChange +from ably.pubsub.types.presence import PresenceAction, PresenceMessage +from ably.pubsub.util.eventemitter import EventEmitter +from ably.pubsub.util.exceptions import AblyException if TYPE_CHECKING: - from ably.realtime.channel import RealtimeChannel + from ably.pubsub.realtime.channel import RealtimeChannel log = logging.getLogger(__name__) @@ -354,7 +354,7 @@ async def _send_presence(self, presence_messages: list[dict]) -> None: Args: presence_messages: List of encoded presence messages to send """ - from ably.transport.websockettransport import ProtocolMessageAction + from ably.pubsub.transport.websockettransport import ProtocolMessageAction protocol_msg = { 'action': ProtocolMessageAction.PRESENCE, diff --git a/ably/realtime/presencemap.py b/ably/pubsub/realtime/presencemap.py similarity index 99% rename from ably/realtime/presencemap.py rename to ably/pubsub/realtime/presencemap.py index 9c5adace..a5d2c427 100644 --- a/ably/realtime/presencemap.py +++ b/ably/pubsub/realtime/presencemap.py @@ -7,7 +7,7 @@ import logging from typing import Callable, Dict, List, Optional, Tuple -from ably.types.presence import PresenceAction, PresenceMessage +from ably.pubsub.types.presence import PresenceAction, PresenceMessage logger = logging.getLogger(__name__) diff --git a/ably/realtime/realtime.py b/ably/pubsub/realtime/realtime.py similarity index 88% rename from ably/realtime/realtime.py rename to ably/pubsub/realtime/realtime.py index ab435304..36b1118d 100644 --- a/ably/realtime/realtime.py +++ b/ably/pubsub/realtime/realtime.py @@ -2,16 +2,21 @@ import logging from typing import Optional -from ably.realtime.channel import Channels -from ably.realtime.connection import Connection, ConnectionState -from ably.rest.rest import AblyRest +from ably.pubsub.http.http import DefaultPubSubHttpClient +from ably.pubsub.prototypes import PubSubRealtimeClient +from ably.pubsub.realtime.channel import Channels +from ably.pubsub.realtime.connection import Connection, ConnectionState +from ably.pubsub.util.construction import reject_direct_construction log = logging.getLogger(__name__) -class AblyRealtime(AblyRest): +class DefaultPubSubRealtimeClient(DefaultPubSubHttpClient, PubSubRealtimeClient): """ - Ably Realtime Client + The default :class:`~ably.pubsub.prototypes.PubSubRealtimeClient` implementation. + + Built by :func:`ably.pubsub.server.create_realtime_client`, which is the + only way to construct one. Attributes ---------- @@ -35,7 +40,7 @@ class AblyRealtime(AblyRest): """ def __init__(self, key: Optional[str] = None, loop: Optional[asyncio.AbstractEventLoop] = None, **kwargs): - """Constructs a RealtimeClient object using an Ably API key. + """Constructs a PubSubRealtimeClient object using an Ably API key. Parameters ---------- @@ -88,6 +93,7 @@ def __init__(self, key: Optional[str] = None, loop: Optional[asyncio.AbstractEve ValueError If no authentication key is not provided """ + reject_direct_construction(type(self), 'ably.pubsub.server.create_realtime_client') if loop is None: try: diff --git a/ably/rest/__init__.py b/ably/pubsub/request/__init__.py similarity index 100% rename from ably/rest/__init__.py rename to ably/pubsub/request/__init__.py diff --git a/ably/http/http.py b/ably/pubsub/request/http.py similarity index 97% rename from ably/http/http.py rename to ably/pubsub/request/http.py index 6ffd6d36..c664f1ab 100644 --- a/ably/http/http.py +++ b/ably/pubsub/request/http.py @@ -6,12 +6,12 @@ import httpx import msgpack -from ably.http.httputils import HttpUtils -from ably.rest.auth import Auth -from ably.transport.defaults import Defaults -from ably.util.clock import select_clock -from ably.util.exceptions import AblyException -from ably.util.helper import extract_url_params, is_token_error +from ably.pubsub.http.auth import Auth +from ably.pubsub.request.httputils import HttpUtils +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.util.clock import select_clock +from ably.pubsub.util.exceptions import AblyException +from ably.pubsub.util.helper import extract_url_params, is_token_error log = logging.getLogger(__name__) diff --git a/ably/http/httputils.py b/ably/pubsub/request/httputils.py similarity index 90% rename from ably/http/httputils.py rename to ably/pubsub/request/httputils.py index 4bd3e78e..f24a2f00 100644 --- a/ably/http/httputils.py +++ b/ably/pubsub/request/httputils.py @@ -3,7 +3,7 @@ import os import platform -import ably +from ably.pubsub.version import api_version, lib_version class HttpUtils: @@ -53,10 +53,10 @@ def get_host_header(host): @staticmethod def default_headers(version=None): if version is None: - version = ably.api_version + version = api_version return { "X-Ably-Version": version, - "Ably-Agent": f'ably-python/{ably.lib_version} python/{platform.python_version()}' + "Ably-Agent": f'ably-pubsub-python/{lib_version} python/{platform.python_version()}' } @staticmethod diff --git a/ably/http/paginatedresult.py b/ably/pubsub/request/paginatedresult.py similarity index 98% rename from ably/http/paginatedresult.py rename to ably/pubsub/request/paginatedresult.py index 5b8bc158..d3361ee1 100644 --- a/ably/http/paginatedresult.py +++ b/ably/pubsub/request/paginatedresult.py @@ -2,8 +2,8 @@ import logging from urllib.parse import urlencode -from ably.http.http import Request -from ably.util import case +from ably.pubsub.request.http import Request +from ably.pubsub.util import case log = logging.getLogger(__name__) diff --git a/ably/scripts/__init__.py b/ably/pubsub/scripts/__init__.py similarity index 100% rename from ably/scripts/__init__.py rename to ably/pubsub/scripts/__init__.py diff --git a/ably/scripts/unasync.py b/ably/pubsub/scripts/unasync.py similarity index 84% rename from ably/scripts/unasync.py rename to ably/pubsub/scripts/unasync.py index d13e20f2..4f6c8f16 100644 --- a/ably/scripts/unasync.py +++ b/ably/pubsub/scripts/unasync.py @@ -5,7 +5,7 @@ import tokenize_rt rename_classes = [ - "AblyRest", + "DefaultPubSubHttpClient", "Push", "PushAdmin", "Channel", @@ -241,19 +241,32 @@ def run(): _TOKEN_REPLACE["aclose"] = "close" _TOKEN_REPLACE["assert_waiter"] = "assert_waiter_sync" - _IMPORTS_REPLACE["ably"] = "ably.sync" + # Ordering matters: _replace_import takes the first key that is a substring + # of the imported module, and the sync flavour of the public surface lives + # at ably.pubsub.server.sync rather than under ably.pubsub.sync. + _IMPORTS_REPLACE["ably.pubsub.server"] = "ably.pubsub.server.sync" + _IMPORTS_REPLACE["ably.pubsub"] = "ably.pubsub.sync" # here... for class_name in rename_classes: _CLASS_RENAME[class_name] = f"{class_name}Sync" _STRING_REPLACE["Auth"] = "AuthSync" - - src_dir_path = os.path.join(os.getcwd(), "ably") - dest_dir_path = os.path.join(os.getcwd(), "ably", "sync") + # The sync flavour's factory lives in ably.pubsub.server.sync. There is no + # sync realtime factory, so create_realtime_client is left pointing at the + # asynchronous one, which is the right advice for that case. + _STRING_REPLACE['ably.pubsub.server.create_http_client'] = \ + 'ably.pubsub.server.sync.create_http_client' + + src_dir_path = os.path.join(os.getcwd(), "ably", "pubsub") + dest_dir_path = os.path.join(os.getcwd(), "ably", "pubsub", "sync") + # ably/pubsub/server is the hand-written public re-export surface; its + # synchronous flavour lives in ably/pubsub/server/sync.py, not under sync/. + server_dir_path = os.path.join(os.getcwd(), "ably", "pubsub", "server") relevant_src_files = (set(find_files(src_dir_path, "*.py")) - - set(find_files(dest_dir_path, "*.py"))) + set(find_files(dest_dir_path, "*.py")) - + set(find_files(server_dir_path, "*.py"))) unasync_files(list(relevant_src_files), [Rule(fromdir=src_dir_path, todir=dest_dir_path)]) @@ -269,14 +282,16 @@ def run(): _IMPORTS_REPLACE["test.ably"] = "test.ably.sync" _STRING_REPLACE['/../assets/testAppSpec.json'] = '/../../assets/testAppSpec.json' - _STRING_REPLACE['ably.rest.auth.Auth.request_token'] = 'ably.sync.rest.auth.AuthSync.request_token' - _STRING_REPLACE['ably.rest.auth.TokenRequest'] = 'ably.sync.rest.auth.TokenRequest' - _STRING_REPLACE['ably.rest.rest.Http.post'] = 'ably.sync.rest.rest.HttpSync.post' + _STRING_REPLACE['ably.pubsub.http.auth.Auth.request_token'] = \ + 'ably.pubsub.sync.http.auth.AuthSync.request_token' + _STRING_REPLACE['ably.pubsub.http.auth.TokenRequest'] = 'ably.pubsub.sync.http.auth.TokenRequest' + _STRING_REPLACE['ably.pubsub.http.http.Http.post'] = 'ably.pubsub.sync.http.http.HttpSync.post' _STRING_REPLACE['httpx.AsyncClient.send'] = 'httpx.Client.send' - _STRING_REPLACE['ably.util.exceptions.AblyException.raise_for_response'] = \ - 'ably.sync.util.exceptions.AblyException.raise_for_response' - _STRING_REPLACE['ably.rest.rest.AblyRest.time'] = 'ably.sync.rest.rest.AblyRestSync.time' - _STRING_REPLACE['ably.rest.auth.Auth._timestamp'] = 'ably.sync.rest.auth.AuthSync._timestamp' + _STRING_REPLACE['ably.pubsub.util.exceptions.AblyException.raise_for_response'] = \ + 'ably.pubsub.sync.util.exceptions.AblyException.raise_for_response' + _STRING_REPLACE['ably.pubsub.http.http.DefaultPubSubHttpClient.time'] = \ + 'ably.pubsub.sync.http.http.DefaultPubSubHttpClientSync.time' + _STRING_REPLACE['ably.pubsub.http.auth.Auth._timestamp'] = 'ably.pubsub.sync.http.auth.AuthSync._timestamp' # round 1 src_dir_path = os.path.join(os.getcwd(), "test", "ably") @@ -287,8 +302,8 @@ def run(): unasync_files(src_files, [Rule(fromdir=src_dir_path, todir=dest_dir_path)]) # round 2 - src_dir_path = os.path.join(os.getcwd(), "test", "ably", "rest") - dest_dir_path = os.path.join(os.getcwd(), "test", "ably", "sync", "rest") + src_dir_path = os.path.join(os.getcwd(), "test", "ably", "http") + dest_dir_path = os.path.join(os.getcwd(), "test", "ably", "sync", "http") src_files = find_files(src_dir_path, "*.py") unasync_files(src_files, [Rule(fromdir=src_dir_path, todir=dest_dir_path, output_file_prefix="sync_")]) diff --git a/ably/pubsub/server/__init__.py b/ably/pubsub/server/__init__.py new file mode 100644 index 00000000..15c20641 --- /dev/null +++ b/ably/pubsub/server/__init__.py @@ -0,0 +1,299 @@ +"""The Ably Pub/Sub client for servers. + +Servers are trusted environments which typically authenticate with an API key, +and whose connections are exempt from monthly-active-user counting. This +package names that side, so the client an application reaches for is the one +whose package matches where it runs. + +Build a client with :func:`create_http_client` or +:func:`create_realtime_client`; the classes behind them are internal and +reject direct construction. Annotate against the prototypes they return, +:class:`~ably.pubsub.prototypes.PubSubHttpClient` and +:class:`~ably.pubsub.prototypes.PubSubRealtimeClient`. + +This is the public API of the ``ably-pubsub-server`` distribution. Both +``ably`` and ``ably.pubsub`` are PEP 420 namespace packages shared with the +other ``ably-*`` distributions, so neither carries an ``__init__.py`` of its +own and neither exports anything; everything the library offers is reachable +from here, from :mod:`ably.pubsub.server.sync` for the synchronous client, or +from :mod:`ably.pubsub.server.liveobjects` for LiveObjects. +""" + +from ably.pubsub.http.auth import Auth +from ably.pubsub.http.channel import Channels as HttpChannels +from ably.pubsub.http.http import DefaultPubSubHttpClient as _DefaultPubSubHttpClient +from ably.pubsub.http.push import Push +from ably.pubsub.prototypes import PubSubHttpClient, PubSubRealtimeClient +from ably.pubsub.realtime.channel import Channels as RealtimeChannels +from ably.pubsub.realtime.connection import Connection +from ably.pubsub.realtime.realtime import DefaultPubSubRealtimeClient as _DefaultPubSubRealtimeClient +from ably.pubsub.request.paginatedresult import HttpPaginatedResponse, PaginatedResult +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.batch import ( + BatchPresenceFailureResult, + BatchPresenceSuccessResult, + BatchPublishFailureResult, + BatchPublishSpec, + BatchPublishSuccessResult, + BatchResult, +) +from ably.pubsub.types.capability import Capability +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.channelsubscription import PushChannelSubscription +from ably.pubsub.types.device import DeviceDetails +from ably.pubsub.types.message import Message, MessageAction, MessageVersion +from ably.pubsub.types.operations import MessageOperation, PublishResult, UpdateDeleteResult +from ably.pubsub.types.options import Options, VCDiffDecoder +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.construction import factory_construction +from ably.pubsub.util.crypto import CipherParams, generate_random_key, get_default_params +from ably.pubsub.util.exceptions import AblyAuthException, AblyException, IncompatibleClientIdException +from ably.pubsub.vcdiff.defaultvcdiffdecoder import AblyVCDiffDecoder +from ably.pubsub.version import api_version, lib_version + + +def create_http_client(**kwargs) -> PubSubHttpClient: + """Create a server Pub/Sub client that operates entirely over HTTP. + + Handles publish, history, presence reads, stats and token issuing. Give the client one + of `key`, `token` or `token_details` to authenticate, or configure token auth through + `auth_callback` or `auth_url`. + + Parameters + ---------- + **kwargs: client options + **Credentials** + + key: str + A valid Ably API key, in `name:secret` form. Mutually exclusive with `key_name` + and `key_secret`. + token: str + A valid token string to authenticate with in place of an API key. + token_details: TokenDetails + A TokenDetails instance to authenticate with in place of an API key. + key_name: str + The first half of an API key, as an alternative to passing `key`. + key_secret: str + The second half of an API key, as an alternative to passing `key`. + + **Token authentication** + + auth_callback: callable + Called when a new token is needed, and returns a token string, a TokenDetails, + a TokenRequest or a signed token request dict. Prefer this to `auth_url` when + the token source is in-process. + auth_url: str + A URL to request a token from when a new token is needed. The client issues a + request to this URL and uses the response as the token. + auth_method: str + The HTTP method used for the `auth_url` request. Defaults to `GET`. + auth_headers: dict + Headers to include in the `auth_url` request. + auth_params: dict + Query parameters to include in the `auth_url` request. + use_token_auth: bool + Force token authentication even when an API key is available. By default the + client uses basic auth where a key is present. + query_time: bool + Query the Ably service for the current time when signing token requests, rather + than trusting the local clock. Defaults to False. + default_token_params: dict + Parameters applied to every token request this client makes. + + **Connection** + + client_id: str + The ID this client identifies as. Messages it publishes carry this ID, and it + cannot be changed once the client has authenticated. + endpoint: str + Either a routing policy name or a fully qualified domain name to connect to. + Defaults to `main`. Incompatible with `environment`, `rest_host` and + `realtime_host`. + tls: bool + Whether the client should use TLS. Defaults to True. + port: int + The non-TLS port to connect to. Defaults to 80. + tls_port: int + The TLS port to connect to. Defaults to 443. + fallback_hosts: list[str] + Hosts to fall back to when the primary endpoint is unreachable. Set this only + if Ably has provided you with a custom set. + fallback_retry_timeout: float + How long (in milliseconds) a fallback host stays preferred once the client has + failed over to it. The default is 10 minutes (600000 milliseconds). + + **HTTP behaviour** + + http_open_timeout: float + Timeout (in seconds) for establishing an HTTP connection. The default is 4 + seconds. + http_request_timeout: float + Timeout (in seconds) for a complete HTTP request. The default is 10 seconds. + http_max_retry_duration: float + How long (in seconds) the client keeps retrying a request across hosts before + giving up. The default is 15 seconds. + http_max_retry_count: int + The most hosts the client will try for a single request. + + **Protocol** + + use_binary_protocol: bool + Encode messages with MessagePack rather than JSON. Defaults to True. + idempotent_rest_publishing: bool + Assign an ID to published messages so that a retried publish is not duplicated. + Defaults to True. + add_request_ids: bool + Add a `request_id` query parameter to every request, which makes a request + traceable in Ably's logs. Defaults to False. + log_level: int + Verbosity of the library's own logging. Defaults to 0. + + **Deprecated** + + rest_host: str + Deprecated: this option will be removed in a future version. Use `endpoint`. + environment: str + Deprecated: this option will be removed in a future version. Use `endpoint`. + + Returns + ------- + PubSubHttpClient + A client ready to use. It opens no connection until its first request. + + Raises + ------ + ValueError + If `key` is combined with `key_name` or `key_secret`. + AblyException + If no means of authenticating is given, or if `endpoint` is combined with + `environment`, `rest_host` or `realtime_host`. + """ + with factory_construction(): + return _DefaultPubSubHttpClient(**kwargs) + + +def create_realtime_client(**kwargs) -> PubSubRealtimeClient: + """Create a server Pub/Sub client with a persistent realtime connection. + + Everything the HTTP client does, plus subscribing to channels and entering presence. + Accepts every option :func:`create_http_client` does — see there for credentials, + token authentication, connection and HTTP behaviour — and the following in addition. + + Parameters + ---------- + **kwargs: client options + **Connection lifecycle** + + loop: asyncio.AbstractEventLoop + The event loop the client runs on. Defaults to the running loop; the client + logs a warning if it is created outside one. + auto_connect: bool + Connect as soon as the client is created. Set this to False to connect later + with `connect()`. Defaults to True. + recover: str + A recovery key from a previous connection, used to resume that connection's + state and message continuity rather than starting fresh. + queue_messages: bool + Hold messages published while the connection is not yet established and send + them once it is, rather than failing them. Defaults to True. + echo_messages: bool + Deliver the messages this client publishes back to it on its own connection. + LiveObjects writes require it. Defaults to True. + transport_params: dict + Additional query parameters to send when opening the realtime connection. + + **Timeouts and retries** + + realtime_request_timeout: float + How long (in milliseconds) to wait for acknowledgement of an operation + performed over the realtime connection, such as establishing a connection or + sending a HEARTBEAT, CONNECT, ATTACH, DETACH or CLOSE request. The default is + 10 seconds (10000 milliseconds). + disconnected_retry_timeout: float + How long (in milliseconds) to stay DISCONNECTED before attempting to reconnect + automatically. The default is 15 seconds (15000 milliseconds). + suspended_retry_timeout: float + How long (in milliseconds) to stay SUSPENDED before attempting to reconnect + automatically. The default is 30 seconds (30000 milliseconds). + channel_retry_timeout: float + How long (in milliseconds) a channel stays SUSPENDED, following a server + initiated DETACHED, before the client re-attaches it — provided the connection + is CONNECTED. The default is 15 seconds (15000 milliseconds). + connectivity_check_url: str + The URL the client requests to decide whether the internet is reachable. On + failing to reach the primary endpoint, a success response from this URL tells + the client to try a fallback host instead of giving up. + + **Deltas** + + vcdiff_decoder: VCDiffDecoder + The decoder used to apply delta-encoded messages on channels configured with + `params={"delta": "vcdiff"}`. Pass an :class:`AblyVCDiffDecoder`. + + **Deprecated** + + realtime_host: str + Deprecated: this option will be removed in a future version. Use `endpoint`. + + Returns + ------- + PubSubRealtimeClient + A client which, unless `auto_connect=False`, is already connecting. + + Raises + ------ + ValueError + If `key` is combined with `key_name` or `key_secret`. + AblyException + If no means of authenticating is given, or if `endpoint` is combined with + `environment`, `rest_host` or `realtime_host`. + """ + with factory_construction(): + return _DefaultPubSubRealtimeClient(**kwargs) + + +__all__ = [ + 'AblyAuthException', + 'AblyException', + 'AblyVCDiffDecoder', + 'Annotation', + 'AnnotationAction', + 'Auth', + 'BatchPresenceFailureResult', + 'BatchPresenceSuccessResult', + 'BatchPublishFailureResult', + 'BatchPublishSpec', + 'BatchPublishSuccessResult', + 'BatchResult', + 'Capability', + 'ChannelMode', + 'ChannelOptions', + 'CipherParams', + 'Connection', + 'DeviceDetails', + 'HttpChannels', + 'HttpPaginatedResponse', + 'IncompatibleClientIdException', + 'Message', + 'MessageAction', + 'MessageOperation', + 'MessageVersion', + 'Options', + 'PaginatedResult', + 'PubSubHttpClient', + 'PubSubRealtimeClient', + 'PublishResult', + 'Push', + 'PushChannelSubscription', + 'RealtimeChannels', + 'TokenDetails', + 'UpdateDeleteResult', + 'VCDiffDecoder', + 'api_version', + 'create_http_client', + 'create_realtime_client', + 'generate_random_key', + 'get_default_params', + 'lib_version', +] diff --git a/ably/pubsub/server/liveobjects.py b/ably/pubsub/server/liveobjects.py new file mode 100644 index 00000000..de0df2b5 --- /dev/null +++ b/ably/pubsub/server/liveobjects.py @@ -0,0 +1,76 @@ +"""LiveObjects for the Ably Pub/Sub client for servers. + +Shared, conflict-free maps and counters stored on a realtime channel. A channel +reaches them through ``channel.object``, a :class:`RealtimeObject`, once the +client from :func:`ably.pubsub.server.create_realtime_client` has attached it +with the object modes. The names here are the types that API takes and returns, +for building values to write and for annotating what it gives back. +""" + +from ably.pubsub.liveobjects.batch import ( + Batch, + BatchContext, + LiveCounterBatchContext, + LiveMapBatchContext, + PrimitiveBatchContext, +) +from ably.pubsub.liveobjects.enums import ObjectsEvent, ValueType +from ably.pubsub.liveobjects.instance import ( + Instance, + InstanceSubscriptionEvent, + LiveCounterInstance, + LiveMapInstance, + PrimitiveInstance, +) +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction, ObjectsMapSemantics +from ably.pubsub.liveobjects.pathobject import ( + LiveCounterPathObject, + LiveMapPathObject, + PathObject, + PathObjectSubscriptionEvent, + PrimitivePathObject, +) +from ably.pubsub.liveobjects.publicmessage import ObjectMessage, ObjectOperation +from ably.pubsub.liveobjects.realtimeobject import RealtimeObject +from ably.pubsub.liveobjects.subscription import StatusSubscription, Subscription +from ably.pubsub.liveobjects.valuetypes import ( + JsonObject, + JsonValue, + LiveCounter, + LiveMap, + LiveMapValue, + Primitive, +) + +__all__ = [ + 'Batch', + 'BatchContext', + 'Instance', + 'InstanceSubscriptionEvent', + 'JsonObject', + 'JsonValue', + 'LiveCounter', + 'LiveCounterBatchContext', + 'LiveCounterInstance', + 'LiveCounterPathObject', + 'LiveMap', + 'LiveMapBatchContext', + 'LiveMapInstance', + 'LiveMapPathObject', + 'LiveMapValue', + 'ObjectMessage', + 'ObjectOperation', + 'ObjectOperationAction', + 'ObjectsEvent', + 'ObjectsMapSemantics', + 'PathObject', + 'PathObjectSubscriptionEvent', + 'Primitive', + 'PrimitiveBatchContext', + 'PrimitiveInstance', + 'PrimitivePathObject', + 'RealtimeObject', + 'StatusSubscription', + 'Subscription', + 'ValueType', +] diff --git a/ably/pubsub/server/sync.py b/ably/pubsub/server/sync.py new file mode 100644 index 00000000..3fcc0eec --- /dev/null +++ b/ably/pubsub/server/sync.py @@ -0,0 +1,104 @@ +"""The synchronous flavour of the Ably Pub/Sub client for servers. + +Exposes the same HTTP surface as :mod:`ably.pubsub.server`, generated without +``async``/``await`` so it can be driven from a thread rather than an event +loop. There is no synchronous realtime client, so a server that needs to +subscribe to channels should use :func:`ably.pubsub.server.create_realtime_client` +instead. +""" + +from ably.pubsub.sync.http.auth import AuthSync +from ably.pubsub.sync.http.channel import ChannelsSync as HttpChannelsSync +from ably.pubsub.sync.http.http import DefaultPubSubHttpClientSync as _DefaultPubSubHttpClientSync +from ably.pubsub.sync.http.push import PushSync +from ably.pubsub.sync.prototypes import PubSubHttpClient +from ably.pubsub.sync.request.paginatedresult import HttpPaginatedResponseSync, PaginatedResultSync +from ably.pubsub.sync.types.annotation import Annotation, AnnotationAction +from ably.pubsub.sync.types.batch import ( + BatchPresenceFailureResult, + BatchPresenceSuccessResult, + BatchPublishFailureResult, + BatchPublishSpec, + BatchPublishSuccessResult, + BatchResult, +) +from ably.pubsub.sync.types.capability import Capability +from ably.pubsub.sync.types.channelmode import ChannelMode +from ably.pubsub.sync.types.channeloptions import ChannelOptions +from ably.pubsub.sync.types.channelsubscription import PushChannelSubscription +from ably.pubsub.sync.types.device import DeviceDetails +from ably.pubsub.sync.types.message import Message, MessageAction, MessageVersion +from ably.pubsub.sync.types.operations import MessageOperation, PublishResult, UpdateDeleteResult +from ably.pubsub.sync.types.options import Options, VCDiffDecoder +from ably.pubsub.sync.types.tokendetails import TokenDetails +from ably.pubsub.sync.util.construction import factory_construction +from ably.pubsub.sync.util.crypto import CipherParams, generate_random_key, get_default_params +from ably.pubsub.sync.util.exceptions import AblyAuthException, AblyException, IncompatibleClientIdException +from ably.pubsub.sync.vcdiff.defaultvcdiffdecoder import AblyVCDiffDecoder +from ably.pubsub.version import api_version, lib_version + + +def create_http_client(**kwargs) -> PubSubHttpClient: + """Create a synchronous server Pub/Sub client that operates entirely over HTTP. + + Accepts every option :func:`ably.pubsub.server.create_http_client` does — see there + for credentials, token authentication, connection and HTTP behaviour — and behaves + identically, without the event loop. The realtime-only options have no effect here. + + Returns + ------- + PubSubHttpClient + A client ready to use. It opens no connection until its first request. + + Raises + ------ + ValueError + If `key` is combined with `key_name` or `key_secret`. + AblyException + If no means of authenticating is given, or if `endpoint` is combined with + `environment`, `rest_host` or `realtime_host`. + """ + with factory_construction(): + return _DefaultPubSubHttpClientSync(**kwargs) + + +__all__ = [ + 'AblyAuthException', + 'AblyException', + 'AblyVCDiffDecoder', + 'Annotation', + 'AnnotationAction', + 'AuthSync', + 'BatchPresenceFailureResult', + 'BatchPresenceSuccessResult', + 'BatchPublishFailureResult', + 'BatchPublishSpec', + 'BatchPublishSuccessResult', + 'BatchResult', + 'Capability', + 'ChannelMode', + 'ChannelOptions', + 'CipherParams', + 'DeviceDetails', + 'HttpChannelsSync', + 'HttpPaginatedResponseSync', + 'IncompatibleClientIdException', + 'Message', + 'MessageAction', + 'MessageOperation', + 'MessageVersion', + 'Options', + 'PaginatedResultSync', + 'PubSubHttpClient', + 'PublishResult', + 'PushChannelSubscription', + 'PushSync', + 'TokenDetails', + 'UpdateDeleteResult', + 'VCDiffDecoder', + 'api_version', + 'create_http_client', + 'generate_random_key', + 'get_default_params', + 'lib_version', +] diff --git a/ably/transport/__init__.py b/ably/pubsub/transport/__init__.py similarity index 100% rename from ably/transport/__init__.py rename to ably/pubsub/transport/__init__.py diff --git a/ably/transport/defaults.py b/ably/pubsub/transport/defaults.py similarity index 98% rename from ably/transport/defaults.py rename to ably/pubsub/transport/defaults.py index 40d73e08..ae5310e6 100644 --- a/ably/transport/defaults.py +++ b/ably/pubsub/transport/defaults.py @@ -1,5 +1,5 @@ class Defaults: - protocol_version = "5" + protocol_version = "6" connectivity_check_url = "https://internet-up.ably-realtime.com/is-the-internet-up.txt" endpoint = 'main' diff --git a/ably/transport/websockettransport.py b/ably/pubsub/transport/websockettransport.py similarity index 95% rename from ably/transport/websockettransport.py rename to ably/pubsub/transport/websockettransport.py index dbde5413..785bae46 100644 --- a/ably/transport/websockettransport.py +++ b/ably/pubsub/transport/websockettransport.py @@ -10,13 +10,13 @@ import msgpack -from ably.http.httputils import HttpUtils -from ably.transport.defaults import Defaults -from ably.types.connectiondetails import ConnectionDetails -from ably.types.operations import PublishResult -from ably.util.clock import select_clock -from ably.util.eventemitter import EventEmitter -from ably.util.exceptions import AblyException +from ably.pubsub.request.httputils import HttpUtils +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.types.connectiondetails import ConnectionDetails +from ably.pubsub.types.operations import PublishResult +from ably.pubsub.util.clock import select_clock +from ably.pubsub.util.eventemitter import EventEmitter +from ably.pubsub.util.exceptions import AblyException try: # websockets 15+ preferred imports @@ -30,7 +30,7 @@ from websockets.exceptions import ConnectionClosedOK, WebSocketException if TYPE_CHECKING: - from ably.realtime.connection import ConnectionManager + from ably.pubsub.realtime.connection import ConnectionManager log = logging.getLogger(__name__) @@ -207,7 +207,9 @@ async def on_protocol_message(self, msg): ProtocolMessageAction.MESSAGE, ProtocolMessageAction.PRESENCE, ProtocolMessageAction.ANNOTATION, - ProtocolMessageAction.SYNC + ProtocolMessageAction.SYNC, + ProtocolMessageAction.OBJECT, + ProtocolMessageAction.OBJECT_SYNC, ): self.connection_manager.on_channel_message(msg) diff --git a/ably/types/__init__.py b/ably/pubsub/types/__init__.py similarity index 100% rename from ably/types/__init__.py rename to ably/pubsub/types/__init__.py diff --git a/ably/types/annotation.py b/ably/pubsub/types/annotation.py similarity index 98% rename from ably/types/annotation.py rename to ably/pubsub/types/annotation.py index 66ce5271..98e845a3 100644 --- a/ably/types/annotation.py +++ b/ably/pubsub/types/annotation.py @@ -1,11 +1,11 @@ import logging from enum import IntEnum -from ably.types.mixins import EncodeDataMixin -from ably.types.typedbuffer import TypedBuffer -from ably.util.crypto import CipherData -from ably.util.encoding import encode_data -from ably.util.helper import to_text +from ably.pubsub.types.mixins import EncodeDataMixin +from ably.pubsub.types.typedbuffer import TypedBuffer +from ably.pubsub.util.crypto import CipherData +from ably.pubsub.util.encoding import encode_data +from ably.pubsub.util.helper import to_text log = logging.getLogger(__name__) diff --git a/ably/types/authoptions.py b/ably/pubsub/types/authoptions.py similarity index 98% rename from ably/types/authoptions.py rename to ably/pubsub/types/authoptions.py index 7ee06af7..18f0fb30 100644 --- a/ably/types/authoptions.py +++ b/ably/pubsub/types/authoptions.py @@ -1,4 +1,4 @@ -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException class AuthOptions: diff --git a/ably/pubsub/types/batch.py b/ably/pubsub/types/batch.py new file mode 100644 index 00000000..d6d37735 --- /dev/null +++ b/ably/pubsub/types/batch.py @@ -0,0 +1,208 @@ +from typing import Generic, List, TypeVar + +from ably.pubsub.types.presence import PresenceMessage +from ably.pubsub.util.exceptions import AblyException + +T = TypeVar('T') + + +class BatchResult(Generic[T]): + """The results of a batch operation, one per channel (BAR2).""" + + def __init__(self, success_count: int, failure_count: int, results: List[T]): + self.__success_count = success_count + self.__failure_count = failure_count + self.__results = results + + @property + def success_count(self) -> int: + """The number of successful operations (BAR2a).""" + return self.__success_count + + @property + def failure_count(self) -> int: + """The number of unsuccessful operations (BAR2b).""" + return self.__failure_count + + @property + def results(self) -> List[T]: + """The result of each operation in the batch (BAR2c).""" + return self.__results + + @staticmethod + def from_dict(obj, result_from_dict): + """Create a BatchResult from a response envelope, building each result with `result_from_dict`.""" + return BatchResult( + success_count=obj.get('successCount'), + failure_count=obj.get('failureCount'), + results=[result_from_dict(result) for result in obj.get('results') or []], + ) + + +class BatchPublishSpec: + """The messages a batch publish sends, and the channels it sends every one of them to (BSP2).""" + + def __init__(self, channels, messages): + """ + Args: + channels: The names of the channels to publish the messages to (BSP2a). + messages: The `Message` objects to publish to each channel (BSP2b). + """ + self.__channels = channels + self.__messages = messages + + @property + def channels(self): + return self.__channels + + @property + def messages(self): + return self.__messages + + def as_dict(self, binary=False): + """Convert BatchPublishSpec to the wire format, encoding each message per RSL4.""" + return { + 'channels': list(self.channels), + 'messages': [message.as_dict(binary=binary) for message in self.messages], + } + + @classmethod + def factory(cls, spec): + """A BatchPublishSpec from either an instance or a dict with `channels` and `messages` keys.""" + if isinstance(spec, cls): + return spec + if isinstance(spec, dict): + return cls(channels=spec.get('channels'), messages=spec.get('messages')) + raise TypeError(f'Unexpected {type(spec)} batch publish spec, expected a BatchPublishSpec or a dict') + + +class BatchPublishSuccessResult: + """The result of publishing a spec's messages to one channel (BPR2).""" + + def __init__(self, channel, message_id, serials): + """ + Args: + channel: The name of the channel (BPR2a). + message_id: The id prefix shared by the published messages (BPR2b). + serials: The serial of each published message, or None where a conflation rule discarded it (BPR2c). + """ + self.__channel = channel + self.__message_id = message_id + self.__serials = serials + + @property + def channel(self): + return self.__channel + + @property + def message_id(self): + return self.__message_id + + @property + def serials(self): + return self.__serials + + @staticmethod + def from_dict(obj): + return BatchPublishSuccessResult( + channel=obj.get('channel'), + message_id=obj.get('messageId'), + serials=obj.get('serials') or [], + ) + + +class BatchPublishFailureResult: + """The reason a spec's messages could not be published to one channel (BPF2).""" + + def __init__(self, channel, error): + """ + Args: + channel: The name of the channel (BPF2a). + error: An `AblyException` describing why the publish failed (BPF2b). + """ + self.__channel = channel + self.__error = error + + @property + def channel(self): + return self.__channel + + @property + def error(self): + return self.__error + + @staticmethod + def from_dict(obj): + return BatchPublishFailureResult( + channel=obj.get('channel'), + error=AblyException.from_dict(obj['error']), + ) + + +class BatchPresenceSuccessResult: + """The members present on one channel of a batch presence request (BGR2).""" + + def __init__(self, channel, presence): + """ + Args: + channel: The name of the channel (BGR2a). + presence: A `PresenceMessage` for each member present on the channel (BGR2b). + """ + self.__channel = channel + self.__presence = presence + + @property + def channel(self): + return self.__channel + + @property + def presence(self): + return self.__presence + + @staticmethod + def from_dict(obj): + # The server leaves `presence` out for a channel with no members + return BatchPresenceSuccessResult( + channel=obj.get('channel'), + presence=PresenceMessage.from_encoded_array(obj.get('presence') or []), + ) + + +class BatchPresenceFailureResult: + """The reason the members of one channel of a batch presence request could not be retrieved (BGF2).""" + + def __init__(self, channel, error): + """ + Args: + channel: The name of the channel (BGF2a). + error: An `AblyException` describing why the request failed for the channel (BGF2b). + """ + self.__channel = channel + self.__error = error + + @property + def channel(self): + return self.__channel + + @property + def error(self): + return self.__error + + @staticmethod + def from_dict(obj): + return BatchPresenceFailureResult( + channel=obj.get('channel'), + error=AblyException.from_dict(obj['error']), + ) + + +def batch_publish_result_from_dict(obj): + if obj.get('error') is not None: + return BatchPublishFailureResult.from_dict(obj) + return BatchPublishSuccessResult.from_dict(obj) + + +def batch_presence_result_from_dict(obj): + if obj.get('error') is not None: + return BatchPresenceFailureResult.from_dict(obj) + return BatchPresenceSuccessResult.from_dict(obj) diff --git a/ably/types/capability.py b/ably/pubsub/types/capability.py similarity index 100% rename from ably/types/capability.py rename to ably/pubsub/types/capability.py diff --git a/ably/types/channeldetails.py b/ably/pubsub/types/channeldetails.py similarity index 100% rename from ably/types/channeldetails.py rename to ably/pubsub/types/channeldetails.py diff --git a/ably/types/channelmode.py b/ably/pubsub/types/channelmode.py similarity index 89% rename from ably/types/channelmode.py rename to ably/pubsub/types/channelmode.py index 23ed735c..47a5f9c5 100644 --- a/ably/types/channelmode.py +++ b/ably/pubsub/types/channelmode.py @@ -2,7 +2,7 @@ from enum import Enum -from ably.types.flags import Flag +from ably.pubsub.types.flags import Flag class ChannelMode(int, Enum): @@ -12,6 +12,8 @@ class ChannelMode(int, Enum): PRESENCE_SUBSCRIBE = Flag.PRESENCE_SUBSCRIBE ANNOTATION_PUBLISH = Flag.ANNOTATION_PUBLISH ANNOTATION_SUBSCRIBE = Flag.ANNOTATION_SUBSCRIBE + OBJECT_SUBSCRIBE = Flag.OBJECT_SUBSCRIBE + OBJECT_PUBLISH = Flag.OBJECT_PUBLISH def encode_channel_mode(modes: list[ChannelMode]) -> int: diff --git a/ably/types/channeloptions.py b/ably/pubsub/types/channeloptions.py similarity index 94% rename from ably/types/channeloptions.py rename to ably/pubsub/types/channeloptions.py index 3e5052c6..9f7a99e1 100644 --- a/ably/types/channeloptions.py +++ b/ably/pubsub/types/channeloptions.py @@ -2,9 +2,9 @@ from typing import Any -from ably.types.channelmode import ChannelMode -from ably.util.crypto import CipherParams -from ably.util.exceptions import AblyException +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.util.crypto import CipherParams +from ably.pubsub.util.exceptions import AblyException class ChannelOptions: diff --git a/ably/types/channelstate.py b/ably/pubsub/types/channelstate.py similarity index 89% rename from ably/types/channelstate.py rename to ably/pubsub/types/channelstate.py index dcb68d67..fee06ce2 100644 --- a/ably/types/channelstate.py +++ b/ably/pubsub/types/channelstate.py @@ -2,7 +2,7 @@ from enum import Enum from typing import Optional -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException class ChannelState(str, Enum): diff --git a/ably/types/channelsubscription.py b/ably/pubsub/types/channelsubscription.py similarity index 98% rename from ably/types/channelsubscription.py rename to ably/pubsub/types/channelsubscription.py index b4c0dbf8..9cc386c9 100644 --- a/ably/types/channelsubscription.py +++ b/ably/pubsub/types/channelsubscription.py @@ -1,4 +1,4 @@ -from ably.util import case +from ably.pubsub.util import case class PushChannelSubscription: diff --git a/ably/types/connectiondetails.py b/ably/pubsub/types/connectiondetails.py similarity index 53% rename from ably/types/connectiondetails.py rename to ably/pubsub/types/connectiondetails.py index a281daed..d1b461f7 100644 --- a/ably/types/connectiondetails.py +++ b/ably/pubsub/types/connectiondetails.py @@ -8,13 +8,19 @@ class ConnectionDetails: connection_key: str def __init__(self, connection_state_ttl: int, max_idle_interval: int, - connection_key: str, client_id: str): + connection_key: str, client_id: str, site_code: str = None, + objects_gc_grace_period: int = None): self.connection_state_ttl = connection_state_ttl self.max_idle_interval = max_idle_interval self.connection_key = connection_key self.client_id = client_id + # CD2j: the site the connection reached, under which LiveObjects applies its own operations + self.site_code = site_code + # CD2i: how long tombstoned objects and map entries are kept before release, in milliseconds + self.objects_gc_grace_period = objects_gc_grace_period @staticmethod def from_dict(json_dict: dict): return ConnectionDetails(json_dict.get('connectionStateTtl'), json_dict.get('maxIdleInterval'), - json_dict.get('connectionKey'), json_dict.get('clientId')) + json_dict.get('connectionKey'), json_dict.get('clientId'), + json_dict.get('siteCode'), json_dict.get('objectsGCGracePeriod')) diff --git a/ably/types/connectionerrors.py b/ably/pubsub/types/connectionerrors.py similarity index 84% rename from ably/types/connectionerrors.py rename to ably/pubsub/types/connectionerrors.py index bb2fa1f4..4b2157d8 100644 --- a/ably/types/connectionerrors.py +++ b/ably/pubsub/types/connectionerrors.py @@ -1,5 +1,5 @@ -from ably.types.connectionstate import ConnectionState -from ably.util.exceptions import AblyException +from ably.pubsub.types.connectionstate import ConnectionState +from ably.pubsub.util.exceptions import AblyException ConnectionErrors = { ConnectionState.DISCONNECTED: AblyException( diff --git a/ably/types/connectionstate.py b/ably/pubsub/types/connectionstate.py similarity index 93% rename from ably/types/connectionstate.py rename to ably/pubsub/types/connectionstate.py index ec958358..fde91caa 100644 --- a/ably/types/connectionstate.py +++ b/ably/pubsub/types/connectionstate.py @@ -2,7 +2,7 @@ from enum import Enum from typing import Optional -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException class ConnectionState(str, Enum): diff --git a/ably/types/device.py b/ably/pubsub/types/device.py similarity index 98% rename from ably/types/device.py rename to ably/pubsub/types/device.py index c2b84ee5..3948b751 100644 --- a/ably/types/device.py +++ b/ably/pubsub/types/device.py @@ -1,4 +1,4 @@ -from ably.util import case +from ably.pubsub.util import case DevicePushTransportType = {'fcm', 'gcm', 'apns', 'web'} DevicePlatform = {'android', 'ios', 'browser'} diff --git a/ably/types/flags.py b/ably/pubsub/types/flags.py similarity index 84% rename from ably/types/flags.py rename to ably/pubsub/types/flags.py index 86666019..ff8fe3a0 100644 --- a/ably/types/flags.py +++ b/ably/pubsub/types/flags.py @@ -8,6 +8,7 @@ class Flag(int, Enum): RESUMED = 1 << 2 TRANSIENT = 1 << 4 ATTACH_RESUME = 1 << 5 + HAS_OBJECTS = 1 << 7 # Channel mode flags PRESENCE = 1 << 16 PUBLISH = 1 << 17 @@ -15,6 +16,8 @@ class Flag(int, Enum): PRESENCE_SUBSCRIBE = 1 << 19 ANNOTATION_PUBLISH = 1 << 21 ANNOTATION_SUBSCRIBE = 1 << 22 + OBJECT_SUBSCRIBE = 1 << 24 + OBJECT_PUBLISH = 1 << 25 def has_flag(message_flags: int, flag: Flag): diff --git a/ably/types/message.py b/ably/pubsub/types/message.py similarity index 95% rename from ably/types/message.py rename to ably/pubsub/types/message.py index 2d84350b..ad574154 100644 --- a/ably/types/message.py +++ b/ably/pubsub/types/message.py @@ -1,12 +1,14 @@ +import base64 import logging +import os from enum import IntEnum -from ably.types.mixins import DeltaExtras, EncodeDataMixin -from ably.types.typedbuffer import TypedBuffer -from ably.util.crypto import CipherData -from ably.util.encoding import encode_data -from ably.util.exceptions import AblyException -from ably.util.helper import to_text +from ably.pubsub.types.mixins import DeltaExtras, EncodeDataMixin +from ably.pubsub.types.typedbuffer import TypedBuffer +from ably.pubsub.util.crypto import CipherData +from ably.pubsub.util.encoding import encode_data +from ably.pubsub.util.exceptions import AblyException +from ably.pubsub.util.helper import to_text log = logging.getLogger(__name__) @@ -391,6 +393,14 @@ def update_inner_message_fields(proto_msg: dict): msg_index = msg_index + 1 +def assign_idempotent_ids(messages): + """Gives every message a library-generated id, unless one of them already has an id (RSL1k1).""" + if all(message.id is None for message in messages): + base_id = base64.urlsafe_b64encode(os.urandom(12)).decode() + for serial, message in enumerate(messages): + message.id = f'{base_id}:{serial}' + + def make_message_response_handler(cipher): def encrypted_message_response_handler(response): messages = response.to_native() diff --git a/ably/types/mixins.py b/ably/pubsub/types/mixins.py similarity index 98% rename from ably/types/mixins.py rename to ably/pubsub/types/mixins.py index c5fad258..976da3b1 100644 --- a/ably/types/mixins.py +++ b/ably/pubsub/types/mixins.py @@ -3,8 +3,8 @@ import json import logging -from ably.util.crypto import CipherData -from ably.util.exceptions import AblyException +from ably.pubsub.util.crypto import CipherData +from ably.pubsub.util.exceptions import AblyException log = logging.getLogger(__name__) diff --git a/ably/types/operations.py b/ably/pubsub/types/operations.py similarity index 100% rename from ably/types/operations.py rename to ably/pubsub/types/operations.py diff --git a/ably/types/options.py b/ably/pubsub/types/options.py similarity index 96% rename from ably/types/options.py rename to ably/pubsub/types/options.py index 5030a7cd..0a4fc932 100644 --- a/ably/types/options.py +++ b/ably/pubsub/types/options.py @@ -2,9 +2,9 @@ import random from abc import ABC, abstractmethod -from ably.transport.defaults import Defaults -from ably.types.authoptions import AuthOptions -from ably.util.exceptions import AblyException +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.types.authoptions import AuthOptions +from ably.pubsub.util.exceptions import AblyException log = logging.getLogger(__name__) @@ -34,8 +34,8 @@ def __init__(self, client_id=None, log_level=0, tls=True, rest_host=None, realti idempotent_rest_publishing=None, loop=None, auto_connect=True, suspended_retry_timeout=None, connectivity_check_url=None, channel_retry_timeout=Defaults.channel_retry_timeout, add_request_ids=False, - vcdiff_decoder: VCDiffDecoder = None, transport_params=None, _test_options=None, - **kwargs): + vcdiff_decoder: VCDiffDecoder = None, transport_params=None, echo_messages=True, + _test_options=None, **kwargs): super().__init__(**kwargs) @@ -81,7 +81,7 @@ def __init__(self, client_id=None, log_level=0, tls=True, rest_host=None, realti ) if idempotent_rest_publishing is None: - from ably import api_version + from ably.pubsub.version import api_version idempotent_rest_publishing = api_version >= '1.2' if environment is not None and endpoint is None: @@ -108,6 +108,7 @@ def __init__(self, client_id=None, log_level=0, tls=True, rest_host=None, realti self.__tls_port = tls_port self.__use_binary_protocol = use_binary_protocol self.__queue_messages = queue_messages + self.__echo_messages = echo_messages self.__recover = recover self.__endpoint = endpoint self.__http_open_timeout = http_open_timeout @@ -189,6 +190,11 @@ def queue_messages(self): def queue_messages(self, value): self.__queue_messages = value + # TO3h + @property + def echo_messages(self): + return self.__echo_messages + @property def recover(self): return self.__recover diff --git a/ably/types/presence.py b/ably/pubsub/types/presence.py similarity index 96% rename from ably/types/presence.py rename to ably/pubsub/types/presence.py index 669f1ede..66cbdea7 100644 --- a/ably/types/presence.py +++ b/ably/pubsub/types/presence.py @@ -1,11 +1,11 @@ from datetime import datetime, timedelta from urllib import parse -from ably.http.paginatedresult import PaginatedResult -from ably.types.mixins import EncodeDataMixin -from ably.types.typedbuffer import TypedBuffer -from ably.util.crypto import CipherData -from ably.util.encoding import encode_data +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.mixins import EncodeDataMixin +from ably.pubsub.types.typedbuffer import TypedBuffer +from ably.pubsub.util.crypto import CipherData +from ably.pubsub.util.encoding import encode_data def _ms_since_epoch(dt): diff --git a/ably/types/stats.py b/ably/pubsub/types/stats.py similarity index 100% rename from ably/types/stats.py rename to ably/pubsub/types/stats.py diff --git a/ably/types/testoptions.py b/ably/pubsub/types/testoptions.py similarity index 93% rename from ably/types/testoptions.py rename to ably/pubsub/types/testoptions.py index bc50f702..4c795411 100644 --- a/ably/types/testoptions.py +++ b/ably/pubsub/types/testoptions.py @@ -13,7 +13,7 @@ class TestOptions: context manager yielding an object supporting `__aiter__`, `send` and `close`. - `clock`: the source of time the client reads, in place of - `ably.util.clock.Clock`. It supplies `now_ms()`, `monotonic_ms()` and + `ably.pubsub.util.clock.Clock`. It supplies `now_ms()`, `monotonic_ms()` and `timer(timeout_ms, callback)`, where `callback` is either a coroutine function or a plain callable and the return value has a `cancel()` method. diff --git a/ably/types/tokendetails.py b/ably/pubsub/types/tokendetails.py similarity index 97% rename from ably/types/tokendetails.py rename to ably/pubsub/types/tokendetails.py index 1980308d..29c7dd24 100644 --- a/ably/types/tokendetails.py +++ b/ably/pubsub/types/tokendetails.py @@ -1,7 +1,7 @@ import json import time -from ably.types.capability import Capability +from ably.pubsub.types.capability import Capability class TokenDetails: diff --git a/ably/types/tokenrequest.py b/ably/pubsub/types/tokenrequest.py similarity index 100% rename from ably/types/tokenrequest.py rename to ably/pubsub/types/tokenrequest.py diff --git a/ably/types/typedbuffer.py b/ably/pubsub/types/typedbuffer.py similarity index 100% rename from ably/types/typedbuffer.py rename to ably/pubsub/types/typedbuffer.py diff --git a/ably/pubsub/util/__init__.py b/ably/pubsub/util/__init__.py new file mode 100644 index 00000000..ac235ae9 --- /dev/null +++ b/ably/pubsub/util/__init__.py @@ -0,0 +1,10 @@ +import logging + +# Silence "no handler" output for every logger in the library. This would +# normally go in the top-level package's __init__.py, but `ably.pubsub` is a +# PEP 420 namespace package and so has none; `ably.pubsub.util` is the one +# module every entry point, async and sync alike, imports on its way in. The +# unasync'd copy of this module runs the same line, hence the guard. +_root_logger = logging.getLogger('ably.pubsub') +if not any(isinstance(handler, logging.NullHandler) for handler in _root_logger.handlers): + _root_logger.addHandler(logging.NullHandler()) diff --git a/ably/util/case.py b/ably/pubsub/util/case.py similarity index 100% rename from ably/util/case.py rename to ably/pubsub/util/case.py diff --git a/ably/util/clock.py b/ably/pubsub/util/clock.py similarity index 97% rename from ably/util/clock.py rename to ably/pubsub/util/clock.py index 00aef3fd..0d4dd0ed 100644 --- a/ably/util/clock.py +++ b/ably/pubsub/util/clock.py @@ -1,7 +1,7 @@ import time from typing import Callable -from ably.util.helper import Timer +from ably.pubsub.util.helper import Timer class Clock: diff --git a/ably/pubsub/util/construction.py b/ably/pubsub/util/construction.py new file mode 100644 index 00000000..f1a0cd94 --- /dev/null +++ b/ably/pubsub/util/construction.py @@ -0,0 +1,41 @@ +"""Enforcement of the factories as the only way to build a client. + +:mod:`ably.pubsub.server` exposes ``create_http_client`` and +``create_realtime_client`` rather than the client classes themselves, so that +the package a client comes from names the side the application runs on. The +classes stay importable for the library's own use and reject construction from +anywhere else. +""" + +from contextlib import contextmanager +from contextvars import ContextVar + +# True for the duration of a factory call. A ContextVar rather than a module +# global so that concurrent tasks cannot observe each other's state. +_in_factory: ContextVar = ContextVar('ably_pubsub_in_factory', default=False) + + +@contextmanager +def factory_construction(): + """Permit client construction within this block. + + For use by Ably's own factory functions only. + """ + token = _in_factory.set(True) + try: + yield + finally: + _in_factory.reset(token) + + +def reject_direct_construction(cls: type, factory: str) -> None: + """Raise unless we are inside a factory call. + + `factory` names the entry point the caller should have reached for, so + that the error says what to do rather than only what went wrong. + """ + if _in_factory.get(): + return + raise TypeError( + f'{cls.__name__} cannot be constructed directly. Use {factory} instead.' + ) diff --git a/ably/util/crypto.py b/ably/pubsub/util/crypto.py similarity index 98% rename from ably/util/crypto.py rename to ably/pubsub/util/crypto.py index 8d8ddfd9..88a62f92 100644 --- a/ably/util/crypto.py +++ b/ably/pubsub/util/crypto.py @@ -7,8 +7,8 @@ except ImportError: from .nocrypto import AES, Random -from ably.types.typedbuffer import TypedBuffer -from ably.util.exceptions import AblyException +from ably.pubsub.types.typedbuffer import TypedBuffer +from ably.pubsub.util.exceptions import AblyException log = logging.getLogger(__name__) diff --git a/ably/util/encoding.py b/ably/pubsub/util/encoding.py similarity index 91% rename from ably/util/encoding.py rename to ably/pubsub/util/encoding.py index 5187aec2..7f4b89da 100644 --- a/ably/util/encoding.py +++ b/ably/pubsub/util/encoding.py @@ -2,8 +2,8 @@ import json from typing import Any -from ably.util.crypto import CipherData -from ably.util.exceptions import AblyException +from ably.pubsub.util.crypto import CipherData +from ably.pubsub.util.exceptions import AblyException def encode_data(data: Any, encoding_array: list, binary: bool = False): diff --git a/ably/util/eventemitter.py b/ably/pubsub/util/eventemitter.py similarity index 99% rename from ably/util/eventemitter.py rename to ably/pubsub/util/eventemitter.py index 74f0beb6..491b0a00 100644 --- a/ably/util/eventemitter.py +++ b/ably/pubsub/util/eventemitter.py @@ -3,7 +3,7 @@ from pyee.asyncio import AsyncIOEventEmitter -from ably.util.helper import is_callable_or_coroutine +from ably.pubsub.util.helper import is_callable_or_coroutine # pyee's event emitter doesn't support attaching a listener to all events # so to patch it, we create a wrapper which uses two event emitters, one diff --git a/ably/util/exceptions.py b/ably/pubsub/util/exceptions.py similarity index 100% rename from ably/util/exceptions.py rename to ably/pubsub/util/exceptions.py diff --git a/ably/util/helper.py b/ably/pubsub/util/helper.py similarity index 98% rename from ably/util/helper.py rename to ably/pubsub/util/helper.py index 4d37d4f4..934a1af1 100644 --- a/ably/util/helper.py +++ b/ably/pubsub/util/helper.py @@ -9,7 +9,7 @@ import msgpack -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException def get_random_id(): diff --git a/ably/util/nocrypto.py b/ably/pubsub/util/nocrypto.py similarity index 54% rename from ably/util/nocrypto.py rename to ably/pubsub/util/nocrypto.py index a66669b3..b6b71265 100644 --- a/ably/util/nocrypto.py +++ b/ably/pubsub/util/nocrypto.py @@ -2,7 +2,7 @@ class InstallPycrypto: def __getattr__(self, name): raise ImportError( - "This requires to install ably with crypto support: pip install 'ably[crypto]'" + "This requires ably-pubsub-server with crypto support: pip install 'ably-pubsub-server[crypto]'" ) diff --git a/ably/util/__init__.py b/ably/pubsub/vcdiff/__init__.py similarity index 100% rename from ably/util/__init__.py rename to ably/pubsub/vcdiff/__init__.py diff --git a/ably/vcdiff/defaultvcdiffdecoder.py b/ably/pubsub/vcdiff/defaultvcdiffdecoder.py similarity index 88% rename from ably/vcdiff/defaultvcdiffdecoder.py rename to ably/pubsub/vcdiff/defaultvcdiffdecoder.py index ae2d3263..07af3b9e 100644 --- a/ably/vcdiff/defaultvcdiffdecoder.py +++ b/ably/pubsub/vcdiff/defaultvcdiffdecoder.py @@ -5,13 +5,13 @@ It implements the VCDiffDecoder interface. Usage: - from ably.vcdiff import AblyVCDiffDecoder, AblyRealtime + from ably.pubsub.server import AblyVCDiffDecoder, ChannelOptions, create_realtime_client # Create VCDiff decoder vcdiff_decoder = AblyVCDiffDecoder() # Create client with decoder - client = AblyRealtime(key="your-key", vcdiff_decoder=vcdiff_decoder) + client = create_realtime_client(key="your-key", vcdiff_decoder=vcdiff_decoder) # Get channel with delta enabled channel = client.channels.get("test", ChannelOptions(params={"delta": "vcdiff"})) @@ -19,8 +19,8 @@ import logging -from ably.types.options import VCDiffDecoder -from ably.util.exceptions import AblyException +from ably.pubsub.types.options import VCDiffDecoder +from ably.pubsub.util.exceptions import AblyException log = logging.getLogger(__name__) diff --git a/ably/pubsub/version.py b/ably/pubsub/version.py new file mode 100644 index 00000000..59f747d4 --- /dev/null +++ b/ably/pubsub/version.py @@ -0,0 +1,9 @@ +"""Version constants for the Ably Pub/Sub library. + +These live in a module of their own rather than in a package ``__init__.py`` +because ``ably.pubsub`` is a PEP 420 namespace package, shared with the other +``ably-pubsub-*`` distributions, and so cannot carry one. +""" + +api_version = '6' +lib_version = '3.1.4' diff --git a/pyproject.toml b/pyproject.toml index 342f92c2..2e0d5eed 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,9 +1,9 @@ [project] -name = "ably" +name = "ably-pubsub-server" version = "3.1.4" -description = "Python REST and Realtime client library SDK for Ably realtime messaging service" +description = "Python REST and Realtime server client library SDK for Ably Pub/Sub" readme = "LONG_DESCRIPTION.rst" -requires-python = ">=3.7" +requires-python = ">=3.8" license = { text = "Apache-2.0" } authors = [ { name = "Ably", email = "support@ably.com" } @@ -15,7 +15,6 @@ classifiers = [ "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", @@ -59,7 +58,7 @@ dev = [ ] [project.scripts] -unasync = "ably.scripts.unasync:run" +unasync = "ably.pubsub.scripts.unasync:run" [project.urls] Homepage = "https://ably.com" @@ -88,7 +87,9 @@ exclude = [ [tool.hatch.build.targets.wheel] ignore-vcs = true -packages = ["ably"] +# `ably` is a PEP 420 namespace package shared with the other ably-* distributions, +# so it must not be shipped with an __init__.py of its own. +only-include = ["ably/pubsub"] [tool.pytest.ini_options] timeout = 30 @@ -102,7 +103,7 @@ explicit = true [tool.ruff] line-length = 115 extend-exclude = [ - "ably/sync", + "ably/pubsub/sync", "test/ably/sync", ] @@ -116,3 +117,10 @@ ignore = [ [tool.ruff.lint.per-file-ignores] "__init__.py" = ["F401"] # imported but unused + +[tool.ruff.lint.isort] +# `ably.pubsub.sync` and `test.ably.sync` only exist once unasync has generated +# them, and the lint job does not generate them. Left to resolve imports itself, +# ruff files those modules as third-party there and first-party here, so the same +# commit lints clean locally and fails in CI. State it instead of inferring it. +known-first-party = ["ably", "test"] diff --git a/test/ably/rest/authheaders_test.py b/test/ably/http/authheaders_test.py similarity index 95% rename from test/ably/rest/authheaders_test.py rename to test/ably/http/authheaders_test.py index 69f0ad1b..ad4ffaa9 100644 --- a/test/ably/rest/authheaders_test.py +++ b/test/ably/http/authheaders_test.py @@ -2,7 +2,7 @@ import respx from httpx import Response -from ably import AblyRest +from ably.pubsub.server import create_http_client @pytest.fixture(scope='session', autouse=True) @@ -22,7 +22,7 @@ async def test_auth_headers_are_only_sent_to_auth_url(method, binary, override): options = {'auth_url': auth_url, 'auth_method': method, 'use_binary_protocol': binary} if not override: options['auth_headers'] = headers - client = AblyRest(**options) + client = create_http_client(**options) auth_route = respx.request(method, auth_url).mock(return_value=Response(200, json={ 'keyName': 'app.key', 'nonce': 'nonce', 'timestamp': 123456789, 'mac': 'signed-request', 'capability': '{"*":["*"]}', diff --git a/test/ably/rest/encoders_test.py b/test/ably/http/encoders_test.py similarity index 91% rename from test/ably/rest/encoders_test.py rename to test/ably/http/encoders_test.py index f8023c5d..24862161 100644 --- a/test/ably/rest/encoders_test.py +++ b/test/ably/http/encoders_test.py @@ -1,23 +1,18 @@ import base64 import json import logging -import sys from unittest import mock +from unittest.mock import AsyncMock import msgpack import pytest -from ably import CipherParams -from ably.types.message import Message -from ably.util.crypto import get_cipher +from ably.pubsub.server import CipherParams +from ably.pubsub.types.message import Message +from ably.pubsub.util.crypto import get_cipher from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase -if sys.version_info >= (3, 8): - from unittest.mock import AsyncMock -else: - from mock import AsyncMock - log = logging.getLogger(__name__) @@ -31,7 +26,7 @@ async def setup(self): async def test_text_utf8(self): channel = self.ably.channels["persisted:publish"] - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', 'foó') _, kwargs = post_mock.call_args assert json.loads(kwargs['body'])['data'] == 'foó' @@ -41,7 +36,7 @@ async def test_str(self): # This test only makes sense for py2 channel = self.ably.channels["persisted:publish"] - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', 'foo') _, kwargs = post_mock.call_args assert json.loads(kwargs['body'])['data'] == 'foo' @@ -50,7 +45,7 @@ async def test_str(self): async def test_with_binary_type(self): channel = self.ably.channels["persisted:publish"] - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', bytearray(b'foo')) _, kwargs = post_mock.call_args raw_data = json.loads(kwargs['body'])['data'] @@ -60,7 +55,7 @@ async def test_with_binary_type(self): async def test_with_bytes_type(self): channel = self.ably.channels["persisted:publish"] - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', b'foo') _, kwargs = post_mock.call_args raw_data = json.loads(kwargs['body'])['data'] @@ -70,7 +65,7 @@ async def test_with_bytes_type(self): async def test_with_json_dict_data(self): channel = self.ably.channels["persisted:publish"] data = {'foó': 'bár'} - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', data) _, kwargs = post_mock.call_args raw_data = json.loads(json.loads(kwargs['body'])['data']) @@ -80,7 +75,7 @@ async def test_with_json_dict_data(self): async def test_with_json_list_data(self): channel = self.ably.channels["persisted:publish"] data = ['foó', 'bár'] - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', data) _, kwargs = post_mock.call_args raw_data = json.loads(json.loads(kwargs['body'])['data']) @@ -161,7 +156,7 @@ def decrypt(self, payload, options=None): async def test_text_utf8(self): channel = self.ably.channels.get("persisted:publish_enc", cipher=self.cipher_params) - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', 'fóo') _, kwargs = post_mock.call_args assert json.loads(kwargs['body'])['encoding'].strip('/') == 'utf-8/cipher+aes-128-cbc/base64' @@ -172,7 +167,7 @@ async def test_str(self): # This test only makes sense for py2 channel = self.ably.channels["persisted:publish"] - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', 'foo') _, kwargs = post_mock.call_args assert json.loads(kwargs['body'])['data'] == 'foo' @@ -182,7 +177,7 @@ async def test_with_binary_type(self): channel = self.ably.channels.get("persisted:publish_enc", cipher=self.cipher_params) - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', bytearray(b'foo')) _, kwargs = post_mock.call_args @@ -195,7 +190,7 @@ async def test_with_json_dict_data(self): channel = self.ably.channels.get("persisted:publish_enc", cipher=self.cipher_params) data = {'foó': 'bár'} - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', data) _, kwargs = post_mock.call_args assert json.loads(kwargs['body'])['encoding'].strip('/') == 'json/utf-8/cipher+aes-128-cbc/base64' @@ -206,7 +201,7 @@ async def test_with_json_list_data(self): channel = self.ably.channels.get("persisted:publish_enc", cipher=self.cipher_params) data = ['foó', 'bár'] - with mock.patch('ably.rest.rest.Http.post', new_callable=AsyncMock) as post_mock: + with mock.patch('ably.pubsub.http.http.Http.post', new_callable=AsyncMock) as post_mock: await channel.publish('event', data) _, kwargs = post_mock.call_args assert json.loads(kwargs['body'])['encoding'].strip('/') == 'json/utf-8/cipher+aes-128-cbc/base64' @@ -269,7 +264,7 @@ def decode(self, data): async def test_text_utf8(self): channel = self.ably.channels["persisted:publish"] - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish('event', 'foó') _, kwargs = post_mock.call_args @@ -279,7 +274,7 @@ async def test_text_utf8(self): async def test_with_binary_type(self): channel = self.ably.channels["persisted:publish"] - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish('event', bytearray(b'foo')) _, kwargs = post_mock.call_args @@ -289,7 +284,7 @@ async def test_with_binary_type(self): async def test_with_json_dict_data(self): channel = self.ably.channels["persisted:publish"] data = {'foó': 'bár'} - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish('event', data) _, kwargs = post_mock.call_args @@ -300,7 +295,7 @@ async def test_with_json_dict_data(self): async def test_with_json_list_data(self): channel = self.ably.channels["persisted:publish"] data = ['foó', 'bár'] - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish('event', data) _, kwargs = post_mock.call_args @@ -367,7 +362,7 @@ def decode(self, data): async def test_text_utf8(self): channel = self.ably.channels.get("persisted:publish_enc", cipher=self.cipher_params) - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish('event', 'fóo') _, kwargs = post_mock.call_args @@ -379,7 +374,7 @@ async def test_with_binary_type(self): channel = self.ably.channels.get("persisted:publish_enc", cipher=self.cipher_params) - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish('event', bytearray(b'foo')) _, kwargs = post_mock.call_args @@ -393,7 +388,7 @@ async def test_with_json_dict_data(self): channel = self.ably.channels.get("persisted:publish_enc", cipher=self.cipher_params) data = {'foó': 'bár'} - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish('event', data) _, kwargs = post_mock.call_args @@ -405,7 +400,7 @@ async def test_with_json_list_data(self): channel = self.ably.channels.get("persisted:publish_enc", cipher=self.cipher_params) data = ['foó', 'bár'] - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish('event', data) _, kwargs = post_mock.call_args diff --git a/test/ably/rest/restannotations_test.py b/test/ably/http/httpannotations_test.py similarity index 97% rename from test/ably/rest/restannotations_test.py rename to test/ably/http/httpannotations_test.py index fcf2c696..3a8ca54b 100644 --- a/test/ably/rest/restannotations_test.py +++ b/test/ably/http/httpannotations_test.py @@ -4,9 +4,9 @@ import pytest -from ably import AblyException -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.message import Message +from ably.pubsub.server import AblyException +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.message import Message from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, assert_waiter @@ -14,7 +14,7 @@ @pytest.mark.parametrize("transport", ["json", "msgpack"], ids=["JSON", "MsgPack"]) -class TestRestAnnotations(BaseAsyncTestCase): +class TestHttpAnnotations(BaseAsyncTestCase): @pytest.fixture(autouse=True) async def setup(self, transport): diff --git a/test/ably/rest/restauth_test.py b/test/ably/http/httpauth_test.py similarity index 92% rename from test/ably/rest/restauth_test.py rename to test/ably/http/httpauth_test.py index 185021e1..3f793565 100644 --- a/test/ably/rest/restauth_test.py +++ b/test/ably/http/httpauth_test.py @@ -1,26 +1,21 @@ import base64 import logging -import sys import time import uuid from unittest import mock +from unittest.mock import AsyncMock from urllib.parse import parse_qs import pytest import respx from httpx import AsyncClient, Response -import ably -from ably import AblyAuthException, AblyRest, Auth -from ably.types.tokendetails import TokenDetails +import ably.pubsub.types.tokenrequest +from ably.pubsub.server import AblyAuthException, Auth, create_http_client +from ably.pubsub.types.tokendetails import TokenDetails from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol -if sys.version_info >= (3, 8): - from unittest.mock import AsyncMock -else: - from mock import AsyncMock - log = logging.getLogger(__name__) @@ -31,19 +26,19 @@ async def setup(self): self.test_vars = await TestApp.get_test_vars() def test_auth_init_key_only(self): - ably = AblyRest(key=self.test_vars["keys"][0]["key_str"]) + ably = create_http_client(key=self.test_vars["keys"][0]["key_str"]) assert Auth.Method.BASIC == ably.auth.auth_mechanism, "Unexpected Auth method mismatch" assert ably.auth.auth_options.key_name == self.test_vars["keys"][0]['key_name'] assert ably.auth.auth_options.key_secret == self.test_vars["keys"][0]['key_secret'] def test_auth_init_token_only(self): - ably = AblyRest(token="this_is_not_really_a_token") + ably = create_http_client(token="this_is_not_really_a_token") assert Auth.Method.TOKEN == ably.auth.auth_mechanism, "Unexpected Auth method mismatch" def test_auth_token_details(self): td = TokenDetails() - ably = AblyRest(token_details=td) + ably = create_http_client(token_details=td) assert Auth.Method.TOKEN == ably.auth.auth_mechanism assert ably.auth.token_details is td @@ -69,7 +64,7 @@ def token_callback(token_params): assert Auth.Method.TOKEN == ably.auth.auth_mechanism, "Unexpected Auth method mismatch" def test_auth_init_with_key_and_client_id(self): - ably = AblyRest(key=self.test_vars["keys"][0]["key_str"], client_id='testClientId') + ably = create_http_client(key=self.test_vars["keys"][0]["key_str"], client_id='testClientId') assert Auth.Method.BASIC == ably.auth.auth_mechanism, "Unexpected Auth method mismatch" assert ably.auth.client_id == 'testClientId' @@ -80,7 +75,7 @@ async def test_auth_init_with_token(self): # RSA11 async def test_request_basic_auth_header(self): - ably = AblyRest(key_secret='foo', key_name='bar') + ably = create_http_client(key_secret='foo', key_name='bar') with mock.patch.object(AsyncClient, 'send') as get_mock: get_mock.return_value = {"status": 200, "headers": {}} @@ -94,7 +89,7 @@ async def test_request_basic_auth_header(self): # RSA7e2 async def test_request_basic_auth_header_with_client_id(self): - ably = AblyRest(key_secret='foo', key_name='bar', client_id='client_id') + ably = create_http_client(key_secret='foo', key_name='bar', client_id='client_id') with mock.patch.object(AsyncClient, 'send') as get_mock: get_mock.return_value = {"status": 200, "headers": {}} @@ -107,7 +102,7 @@ async def test_request_basic_auth_header_with_client_id(self): assert client_id == base64.b64encode('client_id'.encode('ascii')).decode('utf-8') async def test_request_token_auth_header(self): - ably = AblyRest(token='not_a_real_token') + ably = create_http_client(token='not_a_real_token') with mock.patch.object(AsyncClient, 'send') as get_mock: get_mock.return_value = {"status": 200, "headers": {}} @@ -122,26 +117,27 @@ async def test_request_token_auth_header(self): def test_if_cant_authenticate_via_token(self): with pytest.raises(ValueError): - AblyRest(use_token_auth=True) + create_http_client(use_token_auth=True) def test_use_auth_token(self): - ably = AblyRest(use_token_auth=True, key=self.test_vars["keys"][0]["key_str"]) + ably = create_http_client(use_token_auth=True, key=self.test_vars["keys"][0]["key_str"]) assert ably.auth.auth_mechanism == Auth.Method.TOKEN def test_with_client_id(self): - ably = AblyRest(use_token_auth=True, client_id='client_id', key=self.test_vars["keys"][0]["key_str"]) + ably = create_http_client(use_token_auth=True, client_id='client_id', + key=self.test_vars["keys"][0]["key_str"]) assert ably.auth.auth_mechanism == Auth.Method.TOKEN def test_with_auth_url(self): - ably = AblyRest(auth_url='auth_url') + ably = create_http_client(auth_url='auth_url') assert ably.auth.auth_mechanism == Auth.Method.TOKEN def test_with_auth_callback(self): - ably = AblyRest(auth_callback=lambda x: x) + ably = create_http_client(auth_callback=lambda x: x) assert ably.auth.auth_mechanism == Auth.Method.TOKEN def test_with_token(self): - ably = AblyRest(token='a token') + ably = create_http_client(token='a token') assert ably.auth.auth_mechanism == Auth.Method.TOKEN def test_default_ttl_is_1hour(self): @@ -149,19 +145,19 @@ def test_default_ttl_is_1hour(self): assert TokenDetails.DEFAULTS['ttl'] == one_hour_in_ms def test_with_auth_method(self): - ably = AblyRest(token='a token', auth_method='POST') + ably = create_http_client(token='a token', auth_method='POST') assert ably.auth.auth_options.auth_method == 'POST' def test_with_auth_headers(self): - ably = AblyRest(token='a token', auth_headers={'h1': 'v1'}) + ably = create_http_client(token='a token', auth_headers={'h1': 'v1'}) assert ably.auth.auth_options.auth_headers == {'h1': 'v1'} def test_with_auth_params(self): - ably = AblyRest(token='a token', auth_params={'p': 'v'}) + ably = create_http_client(token='a token', auth_params={'p': 'v'}) assert ably.auth.auth_options.auth_params == {'p': 'v'} def test_with_default_token_params(self): - ably = AblyRest(key=self.test_vars["keys"][0]["key_str"], + ably = create_http_client(key=self.test_vars["keys"][0]["key_str"], default_token_params={'ttl': 12345}) assert ably.auth.auth_options.default_token_params == {'ttl': 12345} @@ -198,7 +194,7 @@ async def test_authorize_always_creates_new_token(self): async def test_authorize_create_new_token_if_expired(self): token = await self.ably.auth.authorize() - with mock.patch('ably.rest.auth.Auth.token_details_has_expired', + with mock.patch('ably.pubsub.http.auth.Auth.token_details_has_expired', return_value=True): new_token = await self.ably.auth.authorize() @@ -212,7 +208,7 @@ async def test_authorize_returns_a_token_details(self): async def test_authorize_adheres_to_request_token(self): token_params = {'ttl': 10, 'client_id': 'client_id'} auth_params = {'auth_url': 'somewhere.com', 'query_time': True} - with mock.patch('ably.rest.auth.Auth.request_token', new_callable=AsyncMock) as request_mock: + with mock.patch('ably.pubsub.http.auth.Auth.request_token', new_callable=AsyncMock) as request_mock: await self.ably.auth.authorize(token_params, auth_params) token_called, auth_called = request_mock.call_args @@ -251,7 +247,7 @@ async def test_if_parameters_are_stored_and_used_as_defaults(self): auth_options = dict(self.ably.auth.auth_options.auth_options) auth_options['auth_headers'] = {'a_headers': 'a_value'} await self.ably.auth.authorize({'ttl': 555}, auth_options) - with mock.patch('ably.rest.auth.Auth.request_token', + with mock.patch('ably.pubsub.http.auth.Auth.request_token', wraps=self.ably.auth.request_token) as request_mock: await self.ably.auth.authorize() @@ -263,7 +259,7 @@ async def test_if_parameters_are_stored_and_used_as_defaults(self): auth_options = dict(self.ably.auth.auth_options.auth_options) auth_options['auth_headers'] = None await self.ably.auth.authorize({}, auth_options) - with mock.patch('ably.rest.auth.Auth.request_token', + with mock.patch('ably.pubsub.http.auth.Auth.request_token', wraps=self.ably.auth.request_token) as request_mock: await self.ably.auth.authorize() @@ -283,8 +279,8 @@ async def test_timestamp_is_not_stored(self): # call authorize again with timestamp set timestamp = await self.ably.time() - with mock.patch('ably.rest.auth.TokenRequest', - wraps=ably.types.tokenrequest.TokenRequest) as tr_mock: + with mock.patch('ably.pubsub.http.auth.TokenRequest', + wraps=ably.pubsub.types.tokenrequest.TokenRequest) as tr_mock: auth_options = dict(self.ably.auth.auth_options.auth_options) auth_options['auth_headers'] = {'a_headers': 'a_value'} token_2 = await self.ably.auth.authorize( @@ -295,8 +291,8 @@ async def test_timestamp_is_not_stored(self): assert tr_mock.call_args[1]['timestamp'] == timestamp # call authorize again with no params - with mock.patch('ably.rest.auth.TokenRequest', - wraps=ably.types.tokenrequest.TokenRequest) as tr_mock: + with mock.patch('ably.pubsub.http.auth.TokenRequest', + wraps=ably.pubsub.types.tokenrequest.TokenRequest) as tr_mock: token_4 = await self.ably.auth.authorize() assert isinstance(token_4, TokenDetails) assert token_2 != token_4 diff --git a/test/ably/http/httpbatch_test.py b/test/ably/http/httpbatch_test.py new file mode 100644 index 00000000..f362d5cd --- /dev/null +++ b/test/ably/http/httpbatch_test.py @@ -0,0 +1,112 @@ +import logging + +import pytest + +from ably.pubsub.server import AblyException +from ably.pubsub.types.batch import ( + BatchPresenceSuccessResult, + BatchPublishFailureResult, + BatchPublishSpec, + BatchPublishSuccessResult, + BatchResult, +) +from ably.pubsub.types.message import Message +from test.ably.testapp import TestApp +from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass + +log = logging.getLogger(__name__) + + +class TestHttpBatch(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): + + @pytest.fixture(autouse=True) + async def setup(self): + self.test_vars = await TestApp.get_test_vars() + self.ably = await TestApp.get_ably_rest() + # keys[2] may publish to `canpublish:*` and to nothing else + self.ably_restricted = await TestApp.get_ably_rest(key=self.test_vars['keys'][2]['key_str']) + yield + await self.ably.close() + await self.ably_restricted.close() + + def per_protocol_setup(self, use_binary_protocol): + self.ably.options.use_binary_protocol = use_binary_protocol + self.ably_restricted.options.use_binary_protocol = use_binary_protocol + self.use_binary_protocol = use_binary_protocol + + async def test_batch_publish_single_spec(self): + channel_names = [self.get_channel_name('persisted:batch_publish_a'), + self.get_channel_name('persisted:batch_publish_b')] + + result = await self.ably.batch_publish(BatchPublishSpec(channels=channel_names, messages=[ + Message('string', 'This is a string message payload'), + Message('binary', b'This is a byte[] message payload'), + Message('json', {'test': 'This is a JSONObject message payload'}), + ])) + + assert isinstance(result, BatchResult) + assert result.success_count == 2 + assert result.failure_count == 0 + assert [entry.channel for entry in result.results] == channel_names + for entry in result.results: + assert isinstance(entry, BatchPublishSuccessResult) + assert len(entry.serials) == 3 + + for channel_name in channel_names: + history = await self.ably.channels[channel_name].history(direction='forwards') + messages = history.items + assert [message.name for message in messages] == ['string', 'binary', 'json'] + assert messages[0].data == 'This is a string message payload' + assert bytes(messages[1].data) == b'This is a byte[] message payload' + assert messages[2].data == {'test': 'This is a JSONObject message payload'} + # The library-generated ids share the prefix the result reports + assert [message.id for message in messages] == [f'{result.results[0].message_id}:{serial}' + for serial in range(3)] + + async def test_batch_publish_array_of_specs(self): + channel_a = self.get_channel_name('batch_publish_a') + channel_b = self.get_channel_name('batch_publish_b') + + results = await self.ably.batch_publish([ + BatchPublishSpec(channels=[channel_a], messages=[Message('a', 'data-a')]), + {'channels': [channel_a, channel_b], 'messages': [Message('b', 'data-b')]}, + ]) + + assert isinstance(results, list) + assert len(results) == 2 + assert [entry.channel for entry in results[0].results] == [channel_a] + assert [entry.channel for entry in results[1].results] == [channel_a, channel_b] + assert results[0].results[0].message_id != results[1].results[0].message_id + + async def test_batch_publish_partial_failure(self): + allowed_channel = self.get_channel_name('canpublish:batch_publish') + denied_channel = self.get_channel_name('batch_publish') + + result = await self.ably_restricted.batch_publish(BatchPublishSpec( + channels=[allowed_channel, denied_channel], + messages=[Message('event', 'data')], + )) + + assert result.success_count == 1 + assert result.failure_count == 1 + + success, failure = result.results + assert isinstance(success, BatchPublishSuccessResult) + assert success.channel == allowed_channel + assert isinstance(failure, BatchPublishFailureResult) + assert failure.channel == denied_channel + assert isinstance(failure.error, AblyException) + assert failure.error.code == 40160 + assert failure.error.status_code == 401 + + async def test_batch_presence_empty_channels(self): + channel_names = [self.get_channel_name('batch_presence_a'), self.get_channel_name('batch_presence_b')] + + result = await self.ably.batch_presence(channel_names) + + assert result.success_count == 2 + assert result.failure_count == 0 + assert [entry.channel for entry in result.results] == channel_names + for entry in result.results: + assert isinstance(entry, BatchPresenceSuccessResult) + assert entry.presence == [] diff --git a/test/ably/rest/restcapability_test.py b/test/ably/http/httpcapability_test.py similarity index 98% rename from test/ably/rest/restcapability_test.py rename to test/ably/http/httpcapability_test.py index c95c651d..7e96a57d 100644 --- a/test/ably/rest/restcapability_test.py +++ b/test/ably/http/httpcapability_test.py @@ -1,12 +1,12 @@ import pytest -from ably.types.capability import Capability -from ably.util.exceptions import AblyException +from ably.pubsub.types.capability import Capability +from ably.pubsub.util.exceptions import AblyException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol -class TestRestCapability(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpCapability(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @pytest.fixture(autouse=True) async def setup(self): diff --git a/test/ably/rest/restchannelhistory_test.py b/test/ably/http/httpchannelhistory_test.py similarity index 98% rename from test/ably/rest/restchannelhistory_test.py rename to test/ably/http/httpchannelhistory_test.py index 2b15ae0d..e5773172 100644 --- a/test/ably/rest/restchannelhistory_test.py +++ b/test/ably/http/httpchannelhistory_test.py @@ -3,15 +3,15 @@ import pytest import respx -from ably import AblyException -from ably.http.paginatedresult import PaginatedResult +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.server import AblyException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol log = logging.getLogger(__name__) -class TestRestChannelHistory(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpChannelHistory(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @pytest.fixture(autouse=True) async def setup(self): diff --git a/test/ably/rest/restchannelmutablemessages_test.py b/test/ably/http/httpchannelmutablemessages_test.py similarity index 98% rename from test/ably/rest/restchannelmutablemessages_test.py rename to test/ably/http/httpchannelmutablemessages_test.py index b4f32ef4..e965011c 100644 --- a/test/ably/rest/restchannelmutablemessages_test.py +++ b/test/ably/http/httpchannelmutablemessages_test.py @@ -3,9 +3,9 @@ import pytest -from ably import AblyException, CipherParams, MessageAction -from ably.types.message import Message -from ably.types.operations import MessageOperation +from ably.pubsub.server import AblyException, CipherParams, MessageAction +from ably.pubsub.types.message import Message +from ably.pubsub.types.operations import MessageOperation from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, assert_waiter @@ -13,7 +13,7 @@ @pytest.mark.parametrize("transport", ["json", "msgpack"], ids=["JSON", "MsgPack"]) -class TestRestChannelMutableMessages(BaseAsyncTestCase): +class TestHttpChannelMutableMessages(BaseAsyncTestCase): @pytest.fixture(autouse=True) async def setup(self, transport): diff --git a/test/ably/rest/restchannelpublish_test.py b/test/ably/http/httpchannelpublish_test.py similarity index 97% rename from test/ably/rest/restchannelpublish_test.py rename to test/ably/http/httpchannelpublish_test.py index 5165a00e..1edff7be 100644 --- a/test/ably/rest/restchannelpublish_test.py +++ b/test/ably/http/httpchannelpublish_test.py @@ -10,11 +10,11 @@ import msgpack import pytest -from ably import AblyException, IncompatibleClientIdException, api_version -from ably.rest.auth import Auth -from ably.types.message import Message -from ably.types.tokendetails import TokenDetails -from ably.util import case +from ably.pubsub.http.auth import Auth +from ably.pubsub.server import AblyException, IncompatibleClientIdException, api_version +from ably.pubsub.types.message import Message +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util import case from test.ably import utils from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, assert_waiter, dont_vary_protocol @@ -24,7 +24,7 @@ # Ignore library warning regarding client_id @pytest.mark.filterwarnings('ignore::DeprecationWarning') -class TestRestChannelPublish(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpChannelPublish(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @pytest.fixture(autouse=True) async def setup(self): @@ -103,7 +103,7 @@ async def test_message_list_generate_one_request(self): expected_messages = [Message(f"name-{i}", str(i)) for i in range(3)] - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish(messages=expected_messages) assert post_mock.call_count == 1 @@ -184,7 +184,7 @@ async def test_publish_message_null_name_and_data_keys_arent_sent(self): channel = self.ably.channels[ self.get_channel_name('persisted:null_name_and_data_keys_arent_sent_channel')] - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish(name=None, data=None) @@ -244,7 +244,7 @@ async def test_publish_message_without_client_id_on_identified_client(self): channel = self.ably_with_client_id.channels[ self.get_channel_name('persisted:no_client_id_identified_client')] - with mock.patch('ably.rest.rest.Http.post', + with mock.patch('ably.pubsub.http.http.Http.post', wraps=channel.ably.http.post) as post_mock: await channel.publish(name='publish', data='test') @@ -446,7 +446,7 @@ async def test_publish_params(self): assert 40099 == excinfo.value.code -class TestRestChannelPublishIdempotent(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpChannelPublishIdempotent(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @pytest.fixture(autouse=True) async def setup(self): diff --git a/test/ably/rest/restchannels_test.py b/test/ably/http/httpchannels_test.py similarity index 95% rename from test/ably/rest/restchannels_test.py rename to test/ably/http/httpchannels_test.py index b5e59957..255b23e9 100644 --- a/test/ably/rest/restchannels_test.py +++ b/test/ably/http/httpchannels_test.py @@ -2,9 +2,9 @@ import pytest -from ably import AblyException -from ably.rest.channel import Channel, Channels, Presence -from ably.util.crypto import generate_random_key +from ably.pubsub.http.channel import Channel, Channels, Presence +from ably.pubsub.server import AblyException +from ably.pubsub.util.crypto import generate_random_key from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase diff --git a/test/ably/rest/restchannelstatus_test.py b/test/ably/http/httpchannelstatus_test.py similarity index 97% rename from test/ably/rest/restchannelstatus_test.py rename to test/ably/http/httpchannelstatus_test.py index cb455362..cd027037 100644 --- a/test/ably/rest/restchannelstatus_test.py +++ b/test/ably/http/httpchannelstatus_test.py @@ -8,7 +8,7 @@ log = logging.getLogger(__name__) -class TestRestChannelStatus(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpChannelStatus(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @pytest.fixture(autouse=True) async def setup(self): diff --git a/test/ably/rest/restcrypto_test.py b/test/ably/http/httpcrypto_test.py similarity index 97% rename from test/ably/rest/restcrypto_test.py rename to test/ably/http/httpcrypto_test.py index 94812b29..77b63250 100644 --- a/test/ably/rest/restcrypto_test.py +++ b/test/ably/http/httpcrypto_test.py @@ -6,9 +6,9 @@ import pytest from Crypto import Random -from ably import AblyException -from ably.types.message import Message -from ably.util.crypto import CipherParams, generate_random_key, get_cipher, get_default_params +from ably.pubsub.server import AblyException +from ably.pubsub.types.message import Message +from ably.pubsub.util.crypto import CipherParams, generate_random_key, get_cipher, get_default_params from test.ably import utils from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, BaseTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol @@ -16,7 +16,7 @@ log = logging.getLogger(__name__) -class TestRestCrypto(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpCrypto(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @pytest.fixture(autouse=True) async def setup(self): diff --git a/test/ably/rest/resthttp_test.py b/test/ably/http/httphttp_test.py similarity index 91% rename from test/ably/rest/resthttp_test.py rename to test/ably/http/httphttp_test.py index 385d63ec..7f0bfa04 100644 --- a/test/ably/rest/resthttp_test.py +++ b/test/ably/http/httphttp_test.py @@ -9,16 +9,16 @@ import respx from httpx import Response -from ably import AblyRest -from ably.http.httputils import HttpUtils -from ably.transport.defaults import Defaults -from ably.types.options import Options -from ably.util.exceptions import AblyException +from ably.pubsub.request.httputils import HttpUtils +from ably.pubsub.server import create_http_client +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.types.options import Options +from ably.pubsub.util.exceptions import AblyException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase -class TestRestHttp(BaseAsyncTestCase): +class TestHttpHttp(BaseAsyncTestCase): async def test_host_for_url_brackets_only_ipv6_literals(self): assert HttpUtils.host_for_url('::1') == '[::1]' assert HttpUtils.host_for_url('2001:db8::1') == '[2001:db8::1]' @@ -29,7 +29,7 @@ async def test_host_for_url_brackets_only_ipv6_literals(self): assert HttpUtils.host_for_url('[::1]') == '[::1]' async def test_max_retry_attempts_and_timeouts_defaults(self): - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") assert 'http_open_timeout' in ably.http.CONNECTION_RETRY_DEFAULTS assert 'http_request_timeout' in ably.http.CONNECTION_RETRY_DEFAULTS @@ -42,7 +42,7 @@ async def test_max_retry_attempts_and_timeouts_defaults(self): await ably.close() async def test_cumulative_timeout(self): - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") assert 'http_max_retry_duration' in ably.http.CONNECTION_RETRY_DEFAULTS ably.options.http_max_retry_duration = 0.5 @@ -59,7 +59,7 @@ def sleep_and_raise(*args, **kwargs): await ably.close() async def test_host_fallback(self): - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") def make_url(host): base_url = f"{ably.http.preferred_scheme}://{host}:{ably.http.preferred_port}" @@ -89,7 +89,7 @@ def make_url(host): @respx.mock async def test_no_host_fallback_nor_retries_if_custom_host(self): custom_host = 'example.org' - ably = AblyRest(token="foo", endpoint=custom_host) + ably = create_http_client(token="foo", endpoint=custom_host) mock_route = respx.get("https://example.org").mock(side_effect=httpx.RequestError('')) @@ -139,7 +139,7 @@ async def side_effect(*args, **kwargs): @respx.mock async def test_no_retry_if_not_500_to_599_http_code(self): default_host = Options().get_host() - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") default_url = f"{ably.http.preferred_scheme}://{default_host}:{ably.http.preferred_port}/" @@ -162,7 +162,7 @@ async def test_500_errors(self): https://github.com/ably/ably-python/issues/160 """ - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") mock_request = respx.route().mock(return_value=httpx.Response(500, text="Internal Server Error")) @@ -174,7 +174,7 @@ async def test_500_errors(self): await ably.close() def test_custom_http_timeouts(self): - ably = AblyRest( + ably = create_http_client( token="foo", http_request_timeout=30, http_open_timeout=8, http_max_retry_count=6, http_max_retry_duration=20) @@ -190,11 +190,11 @@ async def test_request_headers(self): # API assert 'X-Ably-Version' in r.request.headers - assert r.request.headers['X-Ably-Version'] == '5' + assert r.request.headers['X-Ably-Version'] == '6' # Agent assert 'Ably-Agent' in r.request.headers - expr = r"^ably-python\/\d+.\d+.\d+(-beta\.\d+)? python\/\d.\d+.\d+$" + expr = r"^ably-pubsub-python\/\d+.\d+.\d+(-beta\.\d+)? python\/\d.\d+.\d+$" assert re.search(expr, r.request.headers['Ably-Agent']) await ably.close() diff --git a/test/ably/rest/restinit_test.py b/test/ably/http/httpinit_test.py similarity index 74% rename from test/ably/rest/restinit_test.py rename to test/ably/http/httpinit_test.py index 25e7c5af..b8202a3b 100644 --- a/test/ably/rest/restinit_test.py +++ b/test/ably/http/httpinit_test.py @@ -3,14 +3,14 @@ import pytest from httpx import AsyncClient -from ably import AblyException, AblyRest -from ably.transport.defaults import Defaults -from ably.types.tokendetails import TokenDetails +from ably.pubsub.server import AblyException, create_http_client +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.types.tokendetails import TokenDetails from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol -class TestRestInit(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpInit(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @pytest.fixture(autouse=True) async def setup(self): @@ -18,7 +18,7 @@ async def setup(self): @dont_vary_protocol def test_key_only(self): - ably = AblyRest(key=self.test_vars["keys"][0]["key_str"]) + ably = create_http_client(key=self.test_vars["keys"][0]["key_str"]) assert ably.options.key_name == self.test_vars["keys"][0]["key_name"], "Key name does not match" assert ably.options.key_secret == self.test_vars["keys"][0]["key_secret"], "Key secret does not match" @@ -27,65 +27,65 @@ def per_protocol_setup(self, use_binary_protocol): @dont_vary_protocol def test_with_token(self): - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") assert ably.options.auth_token == "foo", "Token not set at options" @dont_vary_protocol def test_with_token_details(self): td = TokenDetails() - ably = AblyRest(token_details=td) + ably = create_http_client(token_details=td) assert ably.options.token_details is td @dont_vary_protocol def test_with_options_token_callback(self): def token_callback(**params): return "this_is_not_really_a_token_request" - AblyRest(auth_callback=token_callback) + create_http_client(auth_callback=token_callback) @dont_vary_protocol def test_ambiguous_key_raises_value_error(self): with pytest.raises(ValueError, match="mutually exclusive"): - AblyRest(key=self.test_vars["keys"][0]["key_str"], key_name='x') + create_http_client(key=self.test_vars["keys"][0]["key_str"], key_name='x') with pytest.raises(ValueError, match="mutually exclusive"): - AblyRest(key=self.test_vars["keys"][0]["key_str"], key_secret='x') + create_http_client(key=self.test_vars["keys"][0]["key_str"], key_secret='x') @dont_vary_protocol def test_with_key_name_or_secret_only(self): with pytest.raises(ValueError, match="key is missing"): - AblyRest(key_name='x') + create_http_client(key_name='x') with pytest.raises(ValueError, match="key is missing"): - AblyRest(key_secret='x') + create_http_client(key_secret='x') @dont_vary_protocol def test_with_key_name_and_secret(self): - ably = AblyRest(key_name="foo", key_secret="bar") + ably = create_http_client(key_name="foo", key_secret="bar") assert ably.options.key_name == "foo", "Key name does not match" assert ably.options.key_secret == "bar", "Key secret does not match" @dont_vary_protocol def test_with_options_auth_url(self): - AblyRest(auth_url='not_really_an_url') + create_http_client(auth_url='not_really_an_url') # RSC11 @dont_vary_protocol def test_rest_host_and_environment(self): # endpoint host - ably = AblyRest(token='foo', endpoint="some.other.host") + ably = create_http_client(token='foo', endpoint="some.other.host") assert "some.other.host" == ably.options.get_host(), "Unexpected host mismatch" # endpoint: main - ably = AblyRest(token='foo', endpoint="main") + ably = create_http_client(token='foo', endpoint="main") host = ably.options.get_host() assert "main.realtime.ably.net" == host, f"Unexpected host mismatch {host}" # endpoint: other - ably = AblyRest(token='foo', endpoint="nonprod:sandbox") + ably = create_http_client(token='foo', endpoint="nonprod:sandbox") host = ably.options.get_host() assert "sandbox.realtime.ably-nonprod.net" == host, f"Unexpected host mismatch {host}" # both, as per #TO3k2 with pytest.raises(AblyException): - ably = AblyRest(token='foo', rest_host="some.other.host", + ably = create_http_client(token='foo', rest_host="some.other.host", endpoint="some.other.environment") # RSC15 @@ -99,68 +99,68 @@ def test_fallback_hosts(self): # Fallback hosts specified (RSC15g1) for aux in fallback_hosts: - ably = AblyRest(token='foo', fallback_hosts=aux) + ably = create_http_client(token='foo', fallback_hosts=aux) assert sorted(aux) == sorted(ably.options.get_fallback_hosts()) # Specify endpoint (RSC15g2) - ably = AblyRest(token='foo', endpoint='nonprod:sandbox', http_max_retry_count=10) + ably = create_http_client(token='foo', endpoint='nonprod:sandbox', http_max_retry_count=10) assert sorted(Defaults.get_fallback_hosts('nonprod:sandbox')) == sorted( ably.options.get_fallback_hosts()) # Fallback hosts and endpoint not specified (RSC15g3) - ably = AblyRest(token='foo', http_max_retry_count=10) + ably = create_http_client(token='foo', http_max_retry_count=10) assert sorted(Defaults.get_fallback_hosts()) == sorted(ably.options.get_fallback_hosts()) # RSC15f - ably = AblyRest(token='foo') + ably = create_http_client(token='foo') assert 600000 == ably.options.fallback_retry_timeout - ably = AblyRest(token='foo', fallback_retry_timeout=1000) + ably = create_http_client(token='foo', fallback_retry_timeout=1000) assert 1000 == ably.options.fallback_retry_timeout @dont_vary_protocol def test_specified_host(self): - ably = AblyRest(token='foo', endpoint="some.other.host") + ably = create_http_client(token='foo', endpoint="some.other.host") assert "some.other.host" == ably.options.get_host(), "Unexpected host mismatch" @dont_vary_protocol def test_specified_port(self): - ably = AblyRest(token='foo', port=9998, tls_port=9999) + ably = create_http_client(token='foo', port=9998, tls_port=9999) assert 9999 == Defaults.get_port(ably.options),\ f"Unexpected port mismatch. Expected: 9999. Actual: {ably.options.tls_port}" @dont_vary_protocol def test_specified_non_tls_port(self): - ably = AblyRest(token='foo', port=9998, tls=False) + ably = create_http_client(token='foo', port=9998, tls=False) assert 9998 == Defaults.get_port(ably.options),\ f"Unexpected port mismatch. Expected: 9999. Actual: {ably.options.tls_port}" @dont_vary_protocol def test_specified_tls_port(self): - ably = AblyRest(token='foo', tls_port=9999, tls=True) + ably = create_http_client(token='foo', tls_port=9999, tls=True) assert 9999 == Defaults.get_port(ably.options),\ f"Unexpected port mismatch. Expected: 9999. Actual: {ably.options.tls_port}" @dont_vary_protocol def test_tls_defaults_to_true(self): - ably = AblyRest(token='foo') + ably = create_http_client(token='foo') assert ably.options.tls, "Expected encryption to default to true" assert Defaults.tls_port == Defaults.get_port(ably.options), "Unexpected port mismatch" @dont_vary_protocol def test_tls_can_be_disabled(self): - ably = AblyRest(token='foo', tls=False) + ably = create_http_client(token='foo', tls=False) assert not ably.options.tls, "Expected encryption to be False" assert Defaults.port == Defaults.get_port(ably.options), "Unexpected port mismatch" @dont_vary_protocol def test_with_no_params(self): with pytest.raises(ValueError): - AblyRest() + create_http_client() @dont_vary_protocol def test_with_no_auth_params(self): with pytest.raises(ValueError): - AblyRest(port=111) + create_http_client(port=111) # RSA10k async def test_query_time_param(self): @@ -168,8 +168,8 @@ async def test_query_time_param(self): use_binary_protocol=self.use_binary_protocol) timestamp = ably.auth._timestamp - with patch('ably.rest.rest.AblyRest.time', wraps=ably.time) as server_time,\ - patch('ably.rest.auth.Auth._timestamp', wraps=timestamp) as local_time: + with patch('ably.pubsub.http.http.DefaultPubSubHttpClient.time', wraps=ably.time) as server_time,\ + patch('ably.pubsub.http.auth.Auth._timestamp', wraps=timestamp) as local_time: await ably.auth.request_token() assert local_time.call_count == 1 assert server_time.call_count == 1 @@ -181,19 +181,19 @@ async def test_query_time_param(self): @dont_vary_protocol def test_requests_over_https_production(self): - ably = AblyRest(token='token') + ably = create_http_client(token='token') assert 'https://main.realtime.ably.net' == f'{ably.http.preferred_scheme}://{ably.http.preferred_host}' assert ably.http.preferred_port == 443 @dont_vary_protocol def test_requests_over_http_production(self): - ably = AblyRest(token='token', tls=False) + ably = create_http_client(token='token', tls=False) assert 'http://main.realtime.ably.net' == f'{ably.http.preferred_scheme}://{ ably.http.preferred_host}' assert ably.http.preferred_port == 80 @dont_vary_protocol async def test_request_basic_auth_over_http_fails(self): - ably = AblyRest(key_secret='foo', key_name='bar', tls=False) + ably = create_http_client(key_secret='foo', key_name='bar', tls=False) with pytest.raises(AblyException) as excinfo: await ably.http.get('/time', skip_auth=False) @@ -204,7 +204,7 @@ async def test_request_basic_auth_over_http_fails(self): @dont_vary_protocol async def test_environment(self): - ably = AblyRest(token='token', endpoint='custom') + ably = create_http_client(token='token', endpoint='custom') with patch.object(AsyncClient, 'send', wraps=ably.http._Http__client.send) as get_mock: try: await ably.time() @@ -217,7 +217,7 @@ async def test_environment(self): @dont_vary_protocol def test_accepts_custom_http_timeouts(self): - ably = AblyRest( + ably = create_http_client( token="foo", http_request_timeout=30, http_open_timeout=8, http_max_retry_count=6, http_max_retry_duration=20) diff --git a/test/ably/rest/restpaginatedresult_test.py b/test/ably/http/httppaginatedresult_test.py similarity index 98% rename from test/ably/rest/restpaginatedresult_test.py rename to test/ably/http/httppaginatedresult_test.py index 9aa85689..dd33a071 100644 --- a/test/ably/rest/restpaginatedresult_test.py +++ b/test/ably/http/httppaginatedresult_test.py @@ -2,7 +2,7 @@ import respx from httpx import Response -from ably.http.paginatedresult import PaginatedResult +from ably.pubsub.request.paginatedresult import PaginatedResult from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase diff --git a/test/ably/rest/restpresence_test.py b/test/ably/http/httppresence_test.py similarity index 98% rename from test/ably/rest/restpresence_test.py rename to test/ably/http/httppresence_test.py index 8767b0c6..1fa52d42 100644 --- a/test/ably/rest/restpresence_test.py +++ b/test/ably/http/httppresence_test.py @@ -3,8 +3,8 @@ import pytest import respx -from ably.http.paginatedresult import PaginatedResult -from ably.types.presence import PresenceMessage +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.presence import PresenceMessage from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol diff --git a/test/ably/rest/restpush_test.py b/test/ably/http/httppush_test.py similarity index 98% rename from test/ably/rest/restpush_test.py rename to test/ably/http/httppush_test.py index 30c4798d..9c430cf7 100644 --- a/test/ably/rest/restpush_test.py +++ b/test/ably/http/httppush_test.py @@ -5,8 +5,8 @@ import pytest -from ably import AblyAuthException, AblyException, DeviceDetails, PushChannelSubscription -from ably.http.paginatedresult import PaginatedResult +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.server import AblyAuthException, AblyException, DeviceDetails, PushChannelSubscription from test.ably.testapp import TestApp from test.ably.utils import ( BaseAsyncTestCase, diff --git a/test/ably/rest/restrequest_test.py b/test/ably/http/httprequest_test.py similarity index 96% rename from test/ably/rest/restrequest_test.py rename to test/ably/http/httprequest_test.py index 63c20ccf..ec41c5ff 100644 --- a/test/ably/rest/restrequest_test.py +++ b/test/ably/http/httprequest_test.py @@ -2,15 +2,15 @@ import pytest import respx -from ably import AblyRest -from ably.http.paginatedresult import HttpPaginatedResponse -from ably.transport.defaults import Defaults +from ably.pubsub.request.paginatedresult import HttpPaginatedResponse +from ably.pubsub.server import create_http_client +from ably.pubsub.transport.defaults import Defaults from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol # RSC19 -class TestRestRequest(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpRequest(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @pytest.fixture(autouse=True) async def setup(self): @@ -98,7 +98,7 @@ async def test_headers(self): async def test_timeout(self): # Timeout timeout = 0.000001 - ably = AblyRest(token="foo", http_request_timeout=timeout) + ably = create_http_client(token="foo", http_request_timeout=timeout) assert ably.http.http_request_timeout == timeout # `http_open_timeout` keeps its four-second default, so the request carries a # connect budget as well as a read budget and either can be the one to expire @@ -122,7 +122,7 @@ async def test_timeout(self): await ably.close() # Bad host, no Fallback - ably = AblyRest(key=self.test_vars["keys"][0]["key_str"], + ably = create_http_client(key=self.test_vars["keys"][0]["key_str"], endpoint='some.other.host', port=self.test_vars["port"], tls_port=self.test_vars["tls_port"], diff --git a/test/ably/rest/reststats_test.py b/test/ably/http/httpstats_test.py similarity index 94% rename from test/ably/rest/reststats_test.py rename to test/ably/http/httpstats_test.py index cef28817..5f039058 100644 --- a/test/ably/rest/reststats_test.py +++ b/test/ably/http/httpstats_test.py @@ -3,16 +3,16 @@ import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.types.stats import Stats -from ably.util.exceptions import AblyException +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.stats import Stats +from ably.pubsub.util.exceptions import AblyException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol log = logging.getLogger(__name__) -class TestRestAppStatsSetup: +class TestHttpAppStatsSetup: __stats_added = False def get_params(self): @@ -68,9 +68,9 @@ async def setup(self): } ) # asynctest does not support setUpClass method - if not TestRestAppStatsSetup.__stats_added: + if not TestHttpAppStatsSetup.__stats_added: await self.ably.http.post('/stats', body=stats + previous_stats) - TestRestAppStatsSetup.__stats_added = True + TestHttpAppStatsSetup.__stats_added = True yield await self.ably.close() await self.ably_text.close() @@ -79,7 +79,7 @@ def per_protocol_setup(self, use_binary_protocol): self.ably.options.use_binary_protocol = use_binary_protocol -class TestDirectionForwards(TestRestAppStatsSetup, BaseAsyncTestCase, +class TestDirectionForwards(TestHttpAppStatsSetup, BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): def get_params(self): @@ -105,7 +105,7 @@ async def test_three_pages(self): assert page3.items[0].entries["messages.inbound.realtime.all.count"] == 70 -class TestDirectionBackwards(TestRestAppStatsSetup, BaseAsyncTestCase, +class TestDirectionBackwards(TestHttpAppStatsSetup, BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): def get_params(self): @@ -131,7 +131,7 @@ async def test_three_pages(self): assert page3.items[0].entries["messages.inbound.realtime.all.count"] == 50 -class TestOnlyLastYear(TestRestAppStatsSetup, BaseAsyncTestCase, +class TestOnlyLastYear(TestHttpAppStatsSetup, BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): def get_params(self): @@ -148,7 +148,7 @@ async def test_default_is_backwards(self): assert stats[-1].entries["messages.inbound.realtime.messages.count"] == 50 -class TestPreviousYear(TestRestAppStatsSetup, BaseAsyncTestCase, +class TestPreviousYear(TestHttpAppStatsSetup, BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): def get_params(self): @@ -165,7 +165,7 @@ async def test_default_100_pagination(self): assert len(next_page.items) == 20 -class TestRestAppStats(TestRestAppStatsSetup, BaseAsyncTestCase, +class TestHttpAppStats(TestHttpAppStatsSetup, BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): @dont_vary_protocol diff --git a/test/ably/rest/resttime_test.py b/test/ably/http/httptime_test.py similarity index 92% rename from test/ably/rest/resttime_test.py rename to test/ably/http/httptime_test.py index 4b78620a..5dbeaee2 100644 --- a/test/ably/rest/resttime_test.py +++ b/test/ably/http/httptime_test.py @@ -2,12 +2,12 @@ import pytest -from ably import AblyException +from ably.pubsub.server import AblyException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol -class TestRestTime(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpTime(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): def per_protocol_setup(self, use_binary_protocol): self.ably.options.use_binary_protocol = use_binary_protocol diff --git a/test/ably/rest/resttoken_test.py b/test/ably/http/httptoken_test.py similarity index 91% rename from test/ably/rest/resttoken_test.py rename to test/ably/http/httptoken_test.py index 5052f1be..2406f120 100644 --- a/test/ably/rest/resttoken_test.py +++ b/test/ably/http/httptoken_test.py @@ -5,16 +5,16 @@ import pytest -from ably import AblyException, AblyRest, Capability -from ably.types.tokendetails import TokenDetails -from ably.types.tokenrequest import TokenRequest +from ably.pubsub.server import AblyException, Capability, create_http_client +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.types.tokenrequest import TokenRequest from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, VaryByProtocolTestsMetaclass, dont_vary_protocol log = logging.getLogger(__name__) -class TestRestToken(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): +class TestHttpToken(BaseAsyncTestCase, metaclass=VaryByProtocolTestsMetaclass): async def server_time(self): return await self.ably.time() @@ -120,8 +120,8 @@ async def test_token_generation_with_invalid_ttl(self): async def test_token_generation_with_local_time(self): timestamp = self.ably.auth._timestamp - with patch('ably.rest.rest.AblyRest.time', wraps=self.ably.time) as server_time,\ - patch('ably.rest.auth.Auth._timestamp', wraps=timestamp) as local_time: + with patch('ably.pubsub.http.http.DefaultPubSubHttpClient.time', wraps=self.ably.time) as server_time,\ + patch('ably.pubsub.http.auth.Auth._timestamp', wraps=timestamp) as local_time: await self.ably.auth.request_token() assert local_time.called assert not server_time.called @@ -129,8 +129,8 @@ async def test_token_generation_with_local_time(self): # RSA10k async def test_token_generation_with_server_time(self): timestamp = self.ably.auth._timestamp - with patch('ably.rest.rest.AblyRest.time', wraps=self.ably.time) as server_time,\ - patch('ably.rest.auth.Auth._timestamp', wraps=timestamp) as local_time: + with patch('ably.pubsub.http.http.DefaultPubSubHttpClient.time', wraps=self.ably.time) as server_time,\ + patch('ably.pubsub.http.auth.Auth._timestamp', wraps=timestamp) as local_time: await self.ably.auth.request_token(query_time=True) assert local_time.call_count == 1 assert server_time.call_count == 1 @@ -182,8 +182,8 @@ async def test_key_name_and_secret_are_required(self): @dont_vary_protocol async def test_with_local_time(self): timestamp = self.ably.auth._timestamp - with patch('ably.rest.rest.AblyRest.time', wraps=self.ably.time) as server_time,\ - patch('ably.rest.auth.Auth._timestamp', wraps=timestamp) as local_time: + with patch('ably.pubsub.http.http.DefaultPubSubHttpClient.time', wraps=self.ably.time) as server_time,\ + patch('ably.pubsub.http.auth.Auth._timestamp', wraps=timestamp) as local_time: await self.ably.auth.create_token_request( key_name=self.key_name, key_secret=self.key_secret, query_time=False) assert local_time.called @@ -193,8 +193,8 @@ async def test_with_local_time(self): @dont_vary_protocol async def test_with_server_time(self): timestamp = self.ably.auth._timestamp - with patch('ably.rest.rest.AblyRest.time', wraps=self.ably.time) as server_time,\ - patch('ably.rest.auth.Auth._timestamp', wraps=timestamp) as local_time: + with patch('ably.pubsub.http.http.DefaultPubSubHttpClient.time', wraps=self.ably.time) as server_time,\ + patch('ably.pubsub.http.auth.Auth._timestamp', wraps=timestamp) as local_time: await self.ably.auth.create_token_request( key_name=self.key_name, key_secret=self.key_secret, query_time=True) assert local_time.call_count == 1 @@ -314,7 +314,7 @@ async def auth_callback(token_params): @dont_vary_protocol async def test_hmac(self): - ably = AblyRest(key_name='a_key_name', key_secret='a_secret') + ably = create_http_client(key_name='a_key_name', key_secret='a_secret') token_params = { 'ttl': 1000, 'nonce': 'abcde100', @@ -329,7 +329,7 @@ async def test_hmac(self): # AO2g @dont_vary_protocol async def test_query_server_time(self): - with patch('ably.rest.rest.AblyRest.time', wraps=self.ably.time) as server_time: + with patch('ably.pubsub.http.http.DefaultPubSubHttpClient.time', wraps=self.ably.time) as server_time: await self.ably.auth.create_token_request( key_name=self.key_name, key_secret=self.key_secret, query_time=True) assert server_time.call_count == 1 diff --git a/test/ably/realtime/deltadecoding_test.py b/test/ably/realtime/deltadecoding_test.py index f2930dfb..b067023e 100644 --- a/test/ably/realtime/deltadecoding_test.py +++ b/test/ably/realtime/deltadecoding_test.py @@ -9,12 +9,12 @@ import pytest -from ably import AblyRealtime -from ably.types.channelstate import ChannelState -from ably.types.message import Message -from ably.types.mixins import DecodingContext -from ably.types.options import VCDiffDecoder -from ably.util.exceptions import AblyException +from ably.pubsub.server import create_realtime_client +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.message import Message +from ably.pubsub.types.mixins import DecodingContext +from ably.pubsub.types.options import VCDiffDecoder +from ably.pubsub.util.exceptions import AblyException from test.ably.utils import BaseAsyncTestCase @@ -131,7 +131,7 @@ class TestChannelDeltaBatch(BaseAsyncTestCase): """RealtimeChannel handling of a protocol message carrying several deltas""" def setup_channel(self, decoder): - ably = AblyRealtime(key='not_a.real:key', auto_connect=False, vcdiff_decoder=decoder) + ably = create_realtime_client(key='not_a.real:key', auto_connect=False, vcdiff_decoder=decoder) channel = ably.channels.get('delta') received = [] # subscribe() would also wait for the channel to attach, which this client cannot do diff --git a/test/ably/realtime/eventemitter_test.py b/test/ably/realtime/eventemitter_test.py index 71db2c74..f9913248 100644 --- a/test/ably/realtime/eventemitter_test.py +++ b/test/ably/realtime/eventemitter_test.py @@ -2,7 +2,7 @@ import pytest -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase diff --git a/test/ably/realtime/presencemap_test.py b/test/ably/realtime/presencemap_test.py index 043baeb0..de6002bf 100644 --- a/test/ably/realtime/presencemap_test.py +++ b/test/ably/realtime/presencemap_test.py @@ -8,8 +8,8 @@ import pytest -from ably.realtime.presencemap import PresenceMap, _is_newer -from ably.types.presence import PresenceAction, PresenceMessage +from ably.pubsub.realtime.presencemap import PresenceMap, _is_newer +from ably.pubsub.types.presence import PresenceAction, PresenceMessage from test.ably.utils import BaseAsyncTestCase diff --git a/test/ably/realtime/realtimeannotations_test.py b/test/ably/realtime/realtimeannotations_test.py index 2c5e0658..79521766 100644 --- a/test/ably/realtime/realtimeannotations_test.py +++ b/test/ably/realtime/realtimeannotations_test.py @@ -5,10 +5,10 @@ import pytest -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.channelmode import ChannelMode -from ably.types.channeloptions import ChannelOptions -from ably.types.message import MessageAction +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.message import MessageAction from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, ReusableFuture, assert_waiter @@ -334,7 +334,7 @@ async def on_annotation(annotation): pass # RTAN4e: Should log a warning (not raise), and still register the listener - with caplog.at_level(logging.WARNING, logger='ably.realtime.annotations'): + with caplog.at_level(logging.WARNING, logger='ably.pubsub.realtime.annotations'): await channel.annotations.subscribe(on_annotation) # Verify warning was logged mentioning the missing mode diff --git a/test/ably/realtime/realtimeauth_test.py b/test/ably/realtime/realtimeauth_test.py index a2620489..da073117 100644 --- a/test/ably/realtime/realtimeauth_test.py +++ b/test/ably/realtime/realtimeauth_test.py @@ -5,11 +5,11 @@ import httpx import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.types.connectionstate import ConnectionEvent -from ably.types.tokendetails import TokenDetails +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.connectionstate import ConnectionEvent +from ably.pubsub.types.tokendetails import TokenDetails from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, random_string diff --git a/test/ably/realtime/realtimechannel_publish_test.py b/test/ably/realtime/realtimechannel_publish_test.py index 9ecf10f9..a5af09f5 100644 --- a/test/ably/realtime/realtimechannel_publish_test.py +++ b/test/ably/realtime/realtimechannel_publish_test.py @@ -2,13 +2,13 @@ import pytest -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channeloptions import ChannelOptions -from ably.types.message import Message -from ably.util.crypto import CipherParams -from ably.util.exceptions import AblyException, IncompatibleClientIdException +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.message import Message +from ably.pubsub.util.crypto import CipherParams +from ably.pubsub.util.exceptions import AblyException, IncompatibleClientIdException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, WaitableEvent, assert_waiter @@ -328,7 +328,7 @@ async def test_msgserial_resets_on_new_connection_id(self): new_connection_id = 'new_connection_id_12345' # Simulate server sending CONNECTED with new connectionId - from ably.types.connectiondetails import ConnectionDetails + from ably.pubsub.types.connectiondetails import ConnectionDetails new_connection_details = ConnectionDetails( connection_state_ttl=120000, max_idle_interval=15000, @@ -365,7 +365,7 @@ async def test_msgserial_not_reset_on_same_connection_id(self): # Simulate reconnection with SAME connectionId (transport change, not new connection) same_connection_id = connection_manager.connection_id - from ably.types.connectiondetails import ConnectionDetails + from ably.pubsub.types.connectiondetails import ConnectionDetails connection_details = ConnectionDetails( connection_state_ttl=120000, max_idle_interval=15000, @@ -491,7 +491,7 @@ async def check_pending(): assert connection_manager.msg_serial == 1 # Simulate resume with SAME connectionId - from ably.types.connectiondetails import ConnectionDetails + from ably.pubsub.types.connectiondetails import ConnectionDetails connection_details = ConnectionDetails( connection_state_ttl=120000, max_idle_interval=15000, @@ -541,7 +541,7 @@ async def check_pending(): assert connection_manager.msg_serial == 1 # Simulate NEW connection (different connectionId = failed resume) - from ably.types.connectiondetails import ConnectionDetails + from ably.pubsub.types.connectiondetails import ConnectionDetails new_connection_details = ConnectionDetails( connection_state_ttl=120000, max_idle_interval=15000, diff --git a/test/ably/realtime/realtimechannel_test.py b/test/ably/realtime/realtimechannel_test.py index 6d2865f2..477f412a 100644 --- a/test/ably/realtime/realtimechannel_test.py +++ b/test/ably/realtime/realtimechannel_test.py @@ -2,12 +2,12 @@ import pytest -from ably.realtime.channel import ChannelState, RealtimeChannel -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channeloptions import ChannelOptions -from ably.types.message import Message -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.channel import ChannelState, RealtimeChannel +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.message import Message +from ably.pubsub.util.exceptions import AblyException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, random_string diff --git a/test/ably/realtime/realtimechannel_vcdiff_test.py b/test/ably/realtime/realtimechannel_vcdiff_test.py index 48a484a9..ebc6c7c8 100644 --- a/test/ably/realtime/realtimechannel_vcdiff_test.py +++ b/test/ably/realtime/realtimechannel_vcdiff_test.py @@ -3,10 +3,10 @@ import pytest -from ably import AblyVCDiffDecoder -from ably.realtime.connection import ConnectionState -from ably.types.channeloptions import ChannelOptions -from ably.types.options import VCDiffDecoder +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server import AblyVCDiffDecoder +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.options import VCDiffDecoder from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, WaitableEvent diff --git a/test/ably/realtime/realtimechannelmutablemessages_test.py b/test/ably/realtime/realtimechannelmutablemessages_test.py index afe8a60f..fa4d8db7 100644 --- a/test/ably/realtime/realtimechannelmutablemessages_test.py +++ b/test/ably/realtime/realtimechannelmutablemessages_test.py @@ -3,9 +3,9 @@ import pytest -from ably import AblyException, CipherParams, MessageAction -from ably.types.message import Message -from ably.types.operations import MessageOperation +from ably.pubsub.server import AblyException, CipherParams, MessageAction +from ably.pubsub.types.message import Message +from ably.pubsub.types.operations import MessageOperation from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, WaitableEvent, assert_waiter diff --git a/test/ably/realtime/realtimeconnection_test.py b/test/ably/realtime/realtimeconnection_test.py index d187a378..e6507e6a 100644 --- a/test/ably/realtime/realtimeconnection_test.py +++ b/test/ably/realtime/realtimeconnection_test.py @@ -11,10 +11,10 @@ # websockets 14 and earlier fallback from websockets.server import serve as ws_serve -from ably.realtime.connection import ConnectionEvent, ConnectionState -from ably.transport.defaults import Defaults -from ably.transport.websockettransport import ProtocolMessageAction -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionEvent, ConnectionState +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.util.exceptions import AblyException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase @@ -67,7 +67,7 @@ async def close_active_connection(self): @property def endpoint(self) -> str: - """Host to pass to AblyRealtime (combine with tls=False and port=self.port).""" + """Host to pass to DefaultPubSubRealtimeClient (combine with tls=False and port=self.port).""" return "127.0.0.1" async def __aenter__(self): diff --git a/test/ably/realtime/realtimeinit_test.py b/test/ably/realtime/realtimeinit_test.py index 4009d046..1bc5b7ba 100644 --- a/test/ably/realtime/realtimeinit_test.py +++ b/test/ably/realtime/realtimeinit_test.py @@ -2,9 +2,9 @@ import pytest -from ably import Auth -from ably.realtime.connection import ConnectionState -from ably.util.exceptions import AblyAuthException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server import Auth +from ably.pubsub.util.exceptions import AblyAuthException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase diff --git a/test/ably/realtime/realtimepresence_test.py b/test/ably/realtime/realtimepresence_test.py index 06067f14..63da265d 100644 --- a/test/ably/realtime/realtimepresence_test.py +++ b/test/ably/realtime/realtimepresence_test.py @@ -9,10 +9,10 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.types.channelstate import ChannelState -from ably.types.presence import PresenceAction -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.presence import PresenceAction +from ably.pubsub.util.exceptions import AblyException from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase diff --git a/test/ably/realtime/realtimeresume_test.py b/test/ably/realtime/realtimeresume_test.py index bfe77efa..4f3a89e2 100644 --- a/test/ably/realtime/realtimeresume_test.py +++ b/test/ably/realtime/realtimeresume_test.py @@ -2,9 +2,9 @@ import pytest -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.ably.testapp import TestApp from test.ably.utils import BaseAsyncTestCase, random_string diff --git a/test/ably/testapp.py b/test/ably/testapp.py index 744fd2c7..30a0b6d9 100644 --- a/test/ably/testapp.py +++ b/test/ably/testapp.py @@ -2,12 +2,15 @@ import logging import os -from ably.realtime.realtime import AblyRealtime -from ably.rest.rest import AblyRest -from ably.transport.defaults import Defaults -from ably.types.capability import Capability -from ably.types.options import Options -from ably.util.exceptions import AblyException +# unasync rewrites `from ably.pubsub.server import ...` to the sync package, which +# is HTTP-only. Reaching the realtime factory through the module keeps the +# generated sync copy of this helper importable; it has no realtime tests. +import ably.pubsub.server +from ably.pubsub.server import create_http_client +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.types.capability import Capability +from ably.pubsub.types.options import Options +from ably.pubsub.util.exceptions import AblyException log = logging.getLogger(__name__) @@ -20,10 +23,13 @@ port = 80 tls_port = 443 -ably = AblyRest(token='not_a_real_token', - port=port, tls_port=tls_port, tls=tls, - endpoint=endpoint, - use_binary_protocol=False) +# Not named `ably`: that would shadow the `ably` package imported above, and +# `ably.pubsub.server.create_realtime_client` below would resolve against this +# client instead of the module. +app_setup_client = create_http_client(token='not_a_real_token', + port=port, tls_port=tls_port, tls=tls, + endpoint=endpoint, + use_binary_protocol=False) class TestApp: @@ -32,7 +38,7 @@ class TestApp: @staticmethod async def get_test_vars(): if not TestApp.__test_vars: - r = await ably.http.post("/apps", body=app_spec_local, skip_auth=True) + r = await app_setup_client.http.post("/apps", body=app_spec_local, skip_auth=True) AblyException.raise_for_response(r) app_spec = r.json() @@ -65,7 +71,7 @@ async def get_ably_rest(**kw): test_vars = await TestApp.get_test_vars() options = TestApp.get_options(test_vars, **kw) options.update(kw) - return AblyRest(**options) + return create_http_client(**options) @staticmethod async def get_ably_realtime(**kw): @@ -76,7 +82,7 @@ async def get_ably_realtime(**kw): # for reaching CONNECTED, so a short one keeps a single failed attempt from using # up the whole budget. Tests that assert on retry timing set their own value. options.setdefault('disconnected_retry_timeout', 1000) - return AblyRealtime(**options) + return ably.pubsub.server.create_realtime_client(**options) @staticmethod def get_options(test_vars, **kwargs): diff --git a/test/ably/utils.py b/test/ably/utils.py index a72fdd5c..6715ced7 100644 --- a/test/ably/utils.py +++ b/test/ably/utils.py @@ -11,7 +11,7 @@ import respx from httpx import Response -from ably.http.http import Http +from ably.pubsub.request.http import Http class BaseTestCase: diff --git a/test/unit/annotation_test.py b/test/unit/annotation_test.py index 2cfb0bef..674f14c1 100644 --- a/test/unit/annotation_test.py +++ b/test/unit/annotation_test.py @@ -15,11 +15,11 @@ import pytest -from ably.rest.annotations import construct_validate_annotation, serial_from_msg_or_serial -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.message import Message -from ably.util.crypto import generate_random_key, get_cipher -from ably.util.exceptions import AblyException +from ably.pubsub.http.annotations import construct_validate_annotation, serial_from_msg_or_serial +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.message import Message +from ably.pubsub.util.crypto import generate_random_key, get_cipher +from ably.pubsub.util.exceptions import AblyException # --- RSAN1a3: type validation --- diff --git a/test/unit/batch_test.py b/test/unit/batch_test.py new file mode 100644 index 00000000..8300c3bd --- /dev/null +++ b/test/unit/batch_test.py @@ -0,0 +1,38 @@ +"""Unit tests for the batch types and the inputs the batch methods accept. + +Tests cover: +- BSP2: a dict standing in for a BatchPublishSpec +- RSC22, RSC24: inputs refused before any request is made +""" + +import pytest + +from ably.pubsub.server import create_http_client +from ably.pubsub.types.batch import BatchPublishSpec +from ably.pubsub.types.message import Message + + +def test_batch_publish_spec_factory_accepts_spec(): + spec = BatchPublishSpec(channels=['a'], messages=[Message('event', 'data')]) + assert BatchPublishSpec.factory(spec) is spec + + +def test_batch_publish_spec_factory_accepts_dict(): + messages = [Message('event', 'data')] + spec = BatchPublishSpec.factory({'channels': ['a', 'b'], 'messages': messages}) + assert spec.channels == ['a', 'b'] + assert spec.messages is messages + + +def test_batch_publish_spec_factory_rejects_other_types(): + with pytest.raises(TypeError): + BatchPublishSpec.factory('a') + + +async def test_batch_presence_rejects_a_single_channel_name(): + client = create_http_client(key='fake.key:secret') + try: + with pytest.raises(TypeError): + await client.batch_presence('channel') + finally: + await client.close() diff --git a/test/unit/clock_test.py b/test/unit/clock_test.py index f8814e3b..b02a5fb6 100644 --- a/test/unit/clock_test.py +++ b/test/unit/clock_test.py @@ -1,10 +1,10 @@ import httpx import pytest -from ably import AblyRest -from ably.types.testoptions import TestOptions -from ably.util.clock import Clock -from ably.util.exceptions import AblyException +from ably.pubsub.server import create_http_client +from ably.pubsub.types.testoptions import TestOptions +from ably.pubsub.util.clock import Clock +from ably.pubsub.util.exceptions import AblyException # A plausible wall-clock reading for a fake to start from, so that a time taken # against it looks like a real one. @@ -56,12 +56,12 @@ async def handle_async_request(self, request): def test_auth_timestamps_from_the_injected_clock(): - ably = AblyRest(key='name:secret', _test_options=TestOptions(clock=FakeClock(1_500_000_000_000))) + ably = create_http_client(key='name:secret', _test_options=TestOptions(clock=FakeClock(1_500_000_000_000))) assert ably.auth._timestamp() == 1_500_000_000_000 def test_a_client_without_test_options_reads_the_real_clock(): - ably = AblyRest(token='foo') + ably = create_http_client(token='foo') assert isinstance(ably.auth._Auth__clock, Clock) assert isinstance(ably.http._Http__clock, Clock) @@ -70,8 +70,8 @@ async def test_the_cached_fallback_host_expires_on_the_clock(): # RSC15f: the cache lasts fallback_retry_timeout milliseconds clock = FakeClock() transport = RecordingTransport() - ably = AblyRest(token='foo', fallback_retry_timeout=2000, - _test_options=TestOptions(http_transport=transport, clock=clock)) + ably = create_http_client(token='foo', fallback_retry_timeout=2000, + _test_options=TestOptions(http_transport=transport, clock=clock)) primary = ably.options.get_host() transport.refusing = (primary,) @@ -96,7 +96,7 @@ async def test_every_host_is_tried_while_the_retry_budget_lasts(): # A second of clock time per reading, against the default 15 second budget clock = FakeClock(step_ms=1000) transport = RecordingTransport() - ably = AblyRest(token='foo', _test_options=TestOptions(http_transport=transport, clock=clock)) + ably = create_http_client(token='foo', _test_options=TestOptions(http_transport=transport, clock=clock)) hosts = ably.http.get_hosts() transport.refusing = tuple(hosts) @@ -112,8 +112,8 @@ async def test_retrying_stops_once_the_retry_budget_is_spent(): # RSC15l: http_max_retry_duration is seconds where the clock is milliseconds clock = FakeClock(step_ms=1000) transport = RecordingTransport() - ably = AblyRest(token='foo', http_max_retry_duration=0.5, - _test_options=TestOptions(http_transport=transport, clock=clock)) + ably = create_http_client(token='foo', http_max_retry_duration=0.5, + _test_options=TestOptions(http_transport=transport, clock=clock)) transport.refusing = tuple(ably.http.get_hosts()) with pytest.raises(AblyException): diff --git a/test/unit/http_test.py b/test/unit/http_test.py index 6418dc00..ebae07b2 100644 --- a/test/unit/http_test.py +++ b/test/unit/http_test.py @@ -1,13 +1,13 @@ import httpx import pytest -from ably import AblyRest -from ably.types.testoptions import TestOptions -from ably.util.exceptions import AblyException +from ably.pubsub.server import create_http_client +from ably.pubsub.types.testoptions import TestOptions +from ably.pubsub.util.exceptions import AblyException def test_http_get_rest_hosts_works_when_fallback_realtime_host_is_set(): - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") ably.options.fallback_host = ably.options.get_hosts()[0] # Should not raise TypeError hosts = ably.http.get_hosts() @@ -16,7 +16,7 @@ def test_http_get_rest_hosts_works_when_fallback_realtime_host_is_set(): def test_http_get_rest_hosts_works_when_fallback_realtime_host_is_not_set(): - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") ably.options.fallback_host = None # Should not raise TypeError hosts = ably.http.get_hosts() @@ -36,7 +36,7 @@ async def handle_async_request(self, request): async def test_http_sends_requests_through_an_injected_transport(): transport = RecordingTransport(lambda request: httpx.Response(200, json=[1500000000000])) - ably = AblyRest(token="foo", _test_options=TestOptions(http_transport=transport)) + ably = create_http_client(token="foo", _test_options=TestOptions(http_transport=transport)) server_time = await ably.time() @@ -52,7 +52,7 @@ def refuse(request): raise httpx.ConnectError("connection refused", request=request) transport = RecordingTransport(refuse) - ably = AblyRest(token="foo", _test_options=TestOptions(http_transport=transport)) + ably = create_http_client(token="foo", _test_options=TestOptions(http_transport=transport)) with pytest.raises(AblyException): await ably.time() @@ -63,7 +63,7 @@ def refuse(request): def test_http_uses_a_network_transport_without_test_options(): - ably = AblyRest(token="foo") + ably = create_http_client(token="foo") assert isinstance(ably.http._Http__client._transport, httpx.AsyncHTTPTransport) @@ -74,8 +74,8 @@ def respond(request): return httpx.Response(200, json=[1500000000000]) transport = RecordingTransport(respond) - ably = AblyRest(auth_url='https://auth.example.com/token', - _test_options=TestOptions(http_transport=transport)) + ably = create_http_client(auth_url='https://auth.example.com/token', + _test_options=TestOptions(http_transport=transport)) await ably.auth.authorize() diff --git a/ably/vcdiff/__init__.py b/test/unit/liveobjects/__init__.py similarity index 100% rename from ably/vcdiff/__init__.py rename to test/unit/liveobjects/__init__.py diff --git a/test/unit/liveobjects/batch_test.py b/test/unit/liveobjects/batch_test.py new file mode 100644 index 00000000..fa9b1e0e --- /dev/null +++ b/test/unit/liveobjects/batch_test.py @@ -0,0 +1,698 @@ +"""Batched writes: `batch()` on the typed path and instance views, and the batch contexts. + +Spec points: RTPO20, RTINS17 and RTBC1-RTBC16 (ably/specification#471), with RTO26 as the +precondition a batch checks. No UTS specification covers batch, so these tests are +written against the features specification and LODR-061. + +Each test drives the standard synced pool over the mock websocket, which ACKs every OBJECT +message, so a batch's writes have been applied locally (RTO20) by the time its block exits. +""" + +import json +from typing import NamedTuple + +import pytest + +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction +from ably.pubsub.server.liveobjects import ( + LiveCounter, + LiveCounterBatchContext, + LiveMap, + LiveMapBatchContext, + PrimitiveBatchContext, +) +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import poll_until +from test.uts.helpers.clock import FakeClock, settle +from test.uts.objects.helpers.standard_test_pool import ( + HAS_OBJECTS, + OBJECT, + OBJECT_SUBSCRIBE_FLAG, + build_map_set, + build_object_message, + remote_serial, + setup_synced_channel, + standard_mock_websocket, + time_mock_http, +) + +MAP_CREATE = ObjectOperationAction.MAP_CREATE +MAP_SET = ObjectOperationAction.MAP_SET +MAP_REMOVE = ObjectOperationAction.MAP_REMOVE +COUNTER_CREATE = ObjectOperationAction.COUNTER_CREATE +COUNTER_INC = ObjectOperationAction.COUNTER_INC + +# The server time a test with a fake clock creates objects at +SERVER_TIME_MS = 1_700_000_123_000 + + +async def _synced(**client_options): + """The standard synced channel, and a list of each OBJECT ProtocolMessage the client sends on it.""" + published = [] + mock_ws = standard_mock_websocket(on_object=published.append) + synced = await setup_synced_channel('test', mock_ws=mock_ws, **client_options) + return synced, published + + +def _operations(protocol_message): + """The wire operations of an OBJECT ProtocolMessage the client sent, in order.""" + return [object_message['operation'] for object_message in protocol_message['state']] + + +def _map_set(object_id, key, value): + return {'action': MAP_SET, 'objectId': object_id, 'mapSet': {'key': key, 'value': value}} + + +def _counter_inc(object_id, number): + return {'action': COUNTER_INC, 'objectId': object_id, 'counterInc': {'number': number}} + + +def _assert_error(excinfo, code): + assert excinfo.value.code == code + assert excinfo.value.status_code == 400 + + +# --- Publishing --------------------------------------------------------------------------- + +async def test_rtpo20f_batch_publishes_one_object_message(): + """RTPO20f, RTBC16d: the writes queued in a batch are published in order as one OBJECT + ProtocolMessage, then applied locally.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + ctx.set('name', 'Bob') + ctx.remove('age') + ctx.get('score').as_live_counter().increment(5) + ctx.get('profile').as_live_map().set('email', 'bob@example.com') + + assert len(published) == 1 + assert published[0]['action'] == OBJECT + assert published[0]['channel'] == 'test' + assert _operations(published[0]) == [ + _map_set('root', 'name', {'string': 'Bob'}), + {'action': MAP_REMOVE, 'objectId': 'root', 'mapRemove': {'key': 'age'}}, + _counter_inc('counter:score@1000', 5), + _map_set('map:profile@1000', 'email', {'string': 'bob@example.com'}), + ] + + assert root.get('name').as_primitive().value() == 'Bob' + assert root.get('age').exists() is False + assert root.get('score').as_live_counter().value() == 105 + assert root.at('profile.email').as_primitive().value() == 'bob@example.com' + + +async def test_rtbc12e_nested_value_types_published_in_the_batch(): + """RTBC12e, RTLM20e7g: a `LiveMap` value is published as the creates it evaluates to, nested + ones first, ahead of the MAP_SET that references it, all in the batch's one message. Their + object ids carry the server time (RTO16), which the mock answers from the client's clock.""" + (client, channel, root, mock_ws), published = await _synced(clock=FakeClock(epoch_ms=SERVER_TIME_MS)) + + async with root.batch() as ctx: + ctx.set('team', LiveMap.create({'lead': 'Carol', 'points': LiveCounter.create(10)})) + ctx.get('score').as_live_counter().increment(1) + + assert len(published) == 1 + counter_create, map_create, map_set, counter_inc = _operations(published[0]) + + assert counter_create['action'] == COUNTER_CREATE + assert counter_create['objectId'].startswith('counter:') + assert counter_create['objectId'].endswith(f'@{SERVER_TIME_MS}') + assert json.loads(counter_create['counterCreateWithObjectId']['initialValue']) == {'count': 10} + + assert map_create['action'] == MAP_CREATE + assert map_create['objectId'].startswith('map:') + assert map_create['objectId'].endswith(f'@{SERVER_TIME_MS}') + assert json.loads(map_create['mapCreateWithObjectId']['initialValue'])['entries'] == { + 'lead': {'data': {'string': 'Carol'}}, + 'points': {'data': {'objectId': counter_create['objectId']}}, + } + + assert map_set == _map_set('root', 'team', {'objectId': map_create['objectId']}) + assert counter_inc == _counter_inc('counter:score@1000', 1) + + assert root.get('team').instance().id == map_create['objectId'] + assert root.at('team.lead').as_primitive().value() == 'Carol' + assert root.at('team.points').as_live_counter().value() == 10 + + +async def test_rtbc12e_value_type_set_twice_creates_two_objects(): + """RTBC12e, RTLMV4d1: each write of a `LiveCounter` value is evaluated on its own, so one + value set at two keys creates two counters.""" + (client, channel, root, mock_ws), published = await _synced() + counter = LiveCounter.create(3) + + async with root.batch() as ctx: + ctx.set('first', counter) + ctx.set('second', counter) + + first_create, first_set, second_create, second_set = _operations(published[0]) + assert first_create['objectId'] != second_create['objectId'] + assert first_set == _map_set('root', 'first', {'objectId': first_create['objectId']}) + assert second_set == _map_set('root', 'second', {'objectId': second_create['objectId']}) + assert root.get('first').as_live_counter().value() == 3 + assert root.get('second').as_live_counter().value() == 3 + + +async def test_rtbc16d_empty_batch_publishes_nothing(): + """RTBC16d: a batch that queues nothing publishes nothing.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + assert ctx.get('name').as_primitive().value() == 'Alice' + + assert published == [] + + +async def test_rtbc16b_reads_inside_the_block_see_the_state_before_the_batch(): + """RTBC4-RTBC9, RTBC16b: the writes in a block only queue, so reads inside it resolve against + the objects as they were, and nothing is sent until the block exits.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + score = ctx.get('score').as_live_counter() + ctx.set('name', 'Bob') + ctx.set('city', 'Paris') + ctx.remove('age') + score.increment(5) + + assert ctx.get('name').as_primitive().value() == 'Alice' + assert ctx.get('city') is None + assert ctx.get('age').as_primitive().value() == 30 + assert ctx.keys() == ['name', 'age', 'active', 'score', 'profile', 'data', 'avatar'] + assert ctx.size() == 7 + assert score.value() == 100 + assert root.get('name').as_primitive().value() == 'Alice' + + # Nothing is sent while the block runs, however long it yields to the event loop + await settle() + assert published == [] + + assert root.get('name').as_primitive().value() == 'Bob' + assert root.get('city').as_primitive().value() == 'Paris' + assert root.get('age').exists() is False + assert root.get('score').as_live_counter().value() == 105 + + +async def test_rtpo20g_exception_in_the_block_publishes_nothing(): + """RTPO20g: an exception raised in the block propagates, nothing queued is published, and the + batch is closed.""" + (client, channel, root, mock_ws), published = await _synced() + + with pytest.raises(ValueError, match='abandoned'): + async with root.batch() as ctx: + ctx.set('name', 'Bob') + ctx.get('score').as_live_counter().increment(5) + raise ValueError('abandoned') + + assert published == [] + assert root.get('name').as_primitive().value() == 'Alice' + assert root.get('score').as_live_counter().value() == 100 + + with pytest.raises(AblyException) as excinfo: + ctx.keys() + _assert_error(excinfo, 40000) + + +async def test_rtbc12_invalid_value_inside_a_value_type_raises_at_the_call(): + """RTBC12, RTLMV4c: a `LiveMap` value is evaluated when the batch is flushed, but its + contents are validated when it is set, so an unsupported value inside one raises from the + `set` call, before the server time is fetched, and queues nothing; the batch's other writes + are published.""" + mock_http = time_mock_http() + (client, channel, root, mock_ws), published = await _synced(mock_http=mock_http) + + async with root.batch() as ctx: + with pytest.raises(AblyException) as excinfo: + ctx.set('team', LiveMap.create({'lead': object()})) + ctx.set('name', 'Bob') + + _assert_error(excinfo, 40013) + assert len(published) == 1 + assert _operations(published[0]) == [_map_set('root', 'name', {'string': 'Bob'})] + assert [request for request in mock_http.captured_requests if request.path == '/time'] == [] + + +async def test_rtpo20f_batch_checks_preconditions_again_before_publishing(): + """RTPO20f, RTO26b: a block can await after its writes, so the batch checks the write + preconditions again before publishing; a channel detached in the meantime makes the end of + the block raise 90001, with nothing published.""" + (client, channel, root, mock_ws), published = await _synced() + + with pytest.raises(AblyException) as excinfo: + async with root.batch() as ctx: + ctx.set('name', 'Bob') + await channel.detach() + + _assert_error(excinfo, 90001) + assert published == [] + + +async def test_rtpo20f_rejected_publish_raises_from_the_end_of_the_block(): + """RTPO20f, RTPO20g, RTO20b: when the batch's publish is rejected, the end of the block raises + the rejection's error, nothing is applied locally, and the batch is closed.""" + published = [] + + def reject(message): + published.append(message) + mock_ws.send_to_client({ + 'action': int(ProtocolMessageAction.NACK), + 'msgSerial': message['msgSerial'], + 'count': 1, + 'error': {'code': 40160, 'statusCode': 401, 'message': 'Operation not permitted'}, + }) + + mock_ws = standard_mock_websocket(auto_ack=False, on_object=reject) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + + with pytest.raises(AblyException) as excinfo: + async with root.batch() as ctx: + ctx.set('name', 'Bob') + ctx.get('score').as_live_counter().increment(5) + + assert excinfo.value.code == 40160 + assert len(published) == 1 + assert len(_operations(published[0])) == 2 + assert root.get('name').as_primitive().value() == 'Alice' + assert root.get('score').as_live_counter().value() == 100 + + with pytest.raises(AblyException) as closed: + ctx.keys() + _assert_error(closed, 40000) + + +# --- Entering ----------------------------------------------------------------------------- + +@pytest.mark.parametrize('open_batch', [ + lambda root: root.get('name').as_live_map().batch(), + lambda root: root.get('missing').as_live_map().batch(), + lambda root: root.get('name').get('first').as_live_counter().batch(), + lambda root: root.get('score').as_live_map().batch(), + lambda root: root.as_live_counter().batch(), +], ids=['primitive', 'missing-key', 'below-primitive', 'counter-as-map', 'map-as-counter']) +async def test_rtpo20c_batch_on_a_path_without_the_live_object_raises(open_batch): + """RTPO20c: entering a batch on a path that does not resolve to a live object of the view's + type raises 92007, and publishes nothing.""" + (client, channel, root, mock_ws), published = await _synced() + batch = open_batch(root) + + with pytest.raises(AblyException) as excinfo: + async with batch: + pytest.fail('the batch was entered') + + _assert_error(excinfo, 92007) + assert published == [] + + +async def test_rtpo20c_path_is_resolved_when_the_block_is_entered(): + """RTPO20c: a path batch resolves its path when the block is entered, not when `batch()` is + called.""" + (client, channel, root, mock_ws), published = await _synced() + batch = root.get('visits').as_live_counter().batch() + + await root.set('visits', LiveCounter.create(1)) + async with batch as visits: + assert isinstance(visits, LiveCounterBatchContext) + visits.increment(2) + + assert root.get('visits').as_live_counter().value() == 3 + + +async def test_rtpo20b_batch_requires_publish_mode(): + """RTPO20b, RTO26a: entering a batch on a channel without the OBJECT_PUBLISH mode raises 40024.""" + mock_ws = standard_mock_websocket(attached_flags=HAS_OBJECTS | OBJECT_SUBSCRIBE_FLAG) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws, + modes=(ChannelMode.OBJECT_SUBSCRIBE,)) + + with pytest.raises(AblyException) as excinfo: + async with root.batch(): + pytest.fail('the batch was entered') + + _assert_error(excinfo, 40024) + + +async def test_rtpo20b_batch_requires_echo_messages(): + """RTPO20b, RTO26c: entering a batch on a client with `echo_messages` disabled raises 40000.""" + (client, channel, root, mock_ws), published = await _synced(echo_messages=False) + + with pytest.raises(AblyException) as excinfo: + async with root.batch(): + pytest.fail('the batch was entered') + + _assert_error(excinfo, 40000) + + +async def test_rtpo20b_batch_checks_preconditions_before_resolving(): + """RTPO20b, RTO26b: entering a batch on a detached channel raises 90001, which is checked + before the path is resolved.""" + (client, channel, root, mock_ws), published = await _synced() + score = root.get('score').as_live_counter() + + await channel.detach() + assert channel.state == ChannelState.DETACHED + + # The detach cleared the objects (RTO27a), so `score` no longer resolves: a 92007 here + # would mean the path was resolved first + with pytest.raises(AblyException) as excinfo: + async with score.batch(): + pytest.fail('the batch was entered') + + _assert_error(excinfo, 90001) + + +async def test_rtins17b_instance_batch_checks_preconditions(): + """RTINS17b, RTO26b: entering a batch on an instance of a detached channel raises 90001.""" + (client, channel, root, mock_ws), published = await _synced() + score = root.get('score').instance().as_live_counter() + + await channel.detach() + + with pytest.raises(AblyException) as excinfo: + async with score.batch(): + pytest.fail('the batch was entered') + + _assert_error(excinfo, 90001) + + +async def test_rtpo20d_batch_can_be_entered_once(): + """RTPO20d: each `batch()` call opens one batch, so its block can only be entered once.""" + (client, channel, root, mock_ws), published = await _synced() + batch = root.batch() + + async with batch as ctx: + ctx.set('name', 'Bob') + + with pytest.raises(RuntimeError): + async with batch: + pytest.fail('the batch was entered again') + + assert len(published) == 1 + + +# --- Instances ---------------------------------------------------------------------------- + +async def test_rtins17_batch_on_a_map_instance(): + """RTINS17d-RTINS17f: a batch on a map instance acts on the map it wraps, and is published as + one message.""" + (client, channel, root, mock_ws), published = await _synced() + profile = root.get('profile').instance().as_live_map() + + async with profile.batch() as ctx: + assert isinstance(ctx, LiveMapBatchContext) + assert ctx.id == 'map:profile@1000' + ctx.set('email', 'carol@example.com') + ctx.remove('prefs') + ctx.get('nested_counter').as_live_counter().increment(3) + + assert len(published) == 1 + assert _operations(published[0]) == [ + _map_set('map:profile@1000', 'email', {'string': 'carol@example.com'}), + {'action': MAP_REMOVE, 'objectId': 'map:profile@1000', 'mapRemove': {'key': 'prefs'}}, + _counter_inc('counter:nested@1000', 3), + ] + assert root.at('profile.email').as_primitive().value() == 'carol@example.com' + assert root.at('profile.prefs').exists() is False + assert root.at('profile.nested_counter').as_live_counter().value() == 8 + + +async def test_rtins17_batch_on_a_counter_instance(): + """RTINS17d-RTINS17f: a batch on a counter instance increments the counter it wraps.""" + (client, channel, root, mock_ws), published = await _synced() + score = root.get('score').instance().as_live_counter() + + async with score.batch() as ctx: + assert isinstance(ctx, LiveCounterBatchContext) + ctx.increment(10) + ctx.decrement(4) + + assert _operations(published[0]) == [ + _counter_inc('counter:score@1000', 10), + _counter_inc('counter:score@1000', -4), + ] + assert score.value() == 106 + + +async def test_rtins17d_instance_batch_follows_the_object_not_the_path(): + """RTINS17d, RTPO20c: an instance batch acts on the object the instance wraps wherever it + sits, while a path batch acts on what the path resolves to when it is entered.""" + (client, channel, root, mock_ws), published = await _synced() + profile = root.get('profile').instance().as_live_map() + + await root.remove('profile') + + with pytest.raises(AblyException) as excinfo: + async with root.get('profile').as_live_map().batch(): + pytest.fail('the path batch was entered') + _assert_error(excinfo, 92007) + + async with profile.batch() as ctx: + ctx.set('email', 'dana@example.com') + + assert _operations(published[-1]) == [_map_set('map:profile@1000', 'email', {'string': 'dana@example.com'})] + assert profile.get('email').as_primitive().value() == 'dana@example.com' + + +# --- Counter contexts --------------------------------------------------------------------- + +async def test_rtbc14_rtbc15_counter_batch_increments_and_decrements(): + """RTBC14, RTBC15: a counter context queues a COUNTER_INC for each increment, and one of the + negated amount for each decrement, the amount defaulting to 1.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.get('score').as_live_counter().batch() as score: + assert isinstance(score, LiveCounterBatchContext) + score.increment(5) + score.decrement(2) + score.increment() + score.decrement() + score.increment(0.5) + assert score.value() == 100 + + assert len(published) == 1 + assert _operations(published[0]) == [ + _counter_inc('counter:score@1000', number) for number in (5, -2, 1, -1, 0.5) + ] + assert root.get('score').as_live_counter().value() == 103.5 + + +@pytest.mark.parametrize('write, code', [ + (lambda ctx, score: ctx.set(1, 'value'), 40003), + (lambda ctx, score: ctx.set('key', object()), 40013), + (lambda ctx, score: ctx.set('key', None), 40013), + (lambda ctx, score: ctx.set('key', float('nan')), 40013), + (lambda ctx, score: ctx.remove(None), 40003), + (lambda ctx, score: score.increment('5'), 40003), + (lambda ctx, score: score.increment(None), 40003), + (lambda ctx, score: score.increment(float('inf')), 40003), + (lambda ctx, score: score.decrement(True), 40003), +], ids=['key-not-str', 'value-object', 'value-none', 'value-nan', 'remove-key-none', 'amount-str', + 'amount-none', 'amount-inf', 'amount-bool']) +async def test_rtbc12_rtbc15_invalid_arguments_raise_at_the_call(write, code): + """RTBC12-RTBC15, RTLM20e1, RTLM21e1, RTLC12e1: a write with an invalid key, value or amount + raises where it is called, and queues nothing.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + with pytest.raises(AblyException) as excinfo: + write(ctx, ctx.get('score').as_live_counter()) + _assert_error(excinfo, code) + ctx.set('name', 'Bob') + + assert _operations(published[0]) == [_map_set('root', 'name', {'string': 'Bob'})] + + +# --- Reads and views ---------------------------------------------------------------------- + +async def test_rtbc3_rtbc4_id_and_get(): + """RTBC3, RTBC4: `id` is the wrapped object's id, None for a primitive; `get` wraps the value + at a key in the context of its type, and is None for a key with no value or on anything but + a map.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + assert ctx.id == 'root' + profile, score, name = ctx.get('profile'), ctx.get('score'), ctx.get('name') + assert isinstance(profile, LiveMapBatchContext) + assert isinstance(score, LiveCounterBatchContext) + assert isinstance(name, PrimitiveBatchContext) + assert profile.id == 'map:profile@1000' + assert score.id == 'counter:score@1000' + assert name.id is None + + assert ctx.get('missing') is None + assert name.get('first') is None + assert score.get('first') is None + + with pytest.raises(AblyException) as excinfo: + ctx.get(1) + _assert_error(excinfo, 40003) + + +async def test_rtbc5_value(): + """RTBC5: a counter context's value is the count; a primitive context's is the primitive, + filtered by `expected` as `PrimitiveInstance.value` filters it.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + assert ctx.get('score').as_live_counter().value() == 100 + assert ctx.get('name').as_primitive().value() == 'Alice' + assert ctx.get('name').as_primitive().value(str) == 'Alice' + assert ctx.get('name').as_primitive().value(float) is None + assert ctx.get('active').as_primitive().value(bool) is True + assert ctx.get('active').as_primitive().value(float) is None + assert ctx.get('avatar').as_primitive().value(bytes) == b'\x01\x02\x03' + assert ctx.get('data').as_primitive().value(dict) == {'tags': ['a', 'b']} + + age = ctx.get('age').as_primitive().value(float) + assert age == 30 + assert type(age) is float + + with pytest.raises(TypeError): + ctx.get('age').as_primitive().value(int) + + +async def test_rtbc6_rtbc9_map_enumeration(): + """RTBC6-RTBC9: a map context's entries and values are contexts of each entry's type, its keys + and size those of the map.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.get('profile').as_live_map().batch() as profile: + entries = profile.entries() + assert [key for key, _ in entries] == ['email', 'nested_counter', 'prefs'] + assert [type(context) for _, context in entries] == [ + PrimitiveBatchContext, LiveCounterBatchContext, LiveMapBatchContext] + values = profile.values() + assert [type(context) for context in values] == [type(context) for _, context in entries] + # A live object has one context per batch (RTBC16c); a primitive gets a new one each time + assert values[1:] == [context for _, context in entries[1:]] + assert values[0].value() == 'alice@example.com' + assert profile.keys() == ['email', 'nested_counter', 'prefs'] + assert profile.size() == 3 + assert entries[1][1].value() == 5 + assert entries[2][1].get('theme').as_primitive().value() == 'dark' + + +async def test_rtbc11_compact_json(): + """RTBC11: `compact_json` snapshots the wrapped value as `Instance` does.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + assert ctx.get('profile').compact_json() == { + 'email': 'alice@example.com', 'nested_counter': 5, 'prefs': {'theme': 'dark'}} + assert ctx.compact_json()['avatar'] == 'AQID' + assert ctx.get('score').compact_json() == 100 + + +async def test_rtbc1a_view_helpers_are_checked(): + """RTBC1a, RTTS9d: a context's view helpers return the context itself for the type it wraps, + and raise 92007 for any other.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + score, name = ctx.get('score'), ctx.get('name') + assert ctx.as_live_map() is ctx + assert score.as_live_counter() is score + assert name.as_primitive() is name + + for view_helper in (ctx.as_live_counter, ctx.as_primitive, score.as_live_map, score.as_primitive, + name.as_live_map, name.as_live_counter): + with pytest.raises(AblyException) as excinfo: + view_helper() + _assert_error(excinfo, 92007) + + assert published == [] + + +async def test_rtbc16c_one_context_per_object(): + """RTBC16c: a batch wraps each object in one context however it is reached, the object the + batch was opened on included.""" + (client, channel, root, mock_ws), published = await _synced() + # A reference from `profile.prefs` back to the root, so that the root is reachable from inside + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('map:prefs@1000', 'home', {'objectId': 'root'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: 'home' in root.at('profile.prefs').as_live_map().keys(), + description='the back reference to be applied') + + async with root.batch() as ctx: + profile = ctx.get('profile') + assert ctx.get('profile') is profile + assert dict(ctx.entries())['profile'] is profile + assert ctx.get('score') is ctx.get('score') + assert profile.get('prefs').get('home') is ctx + + +class _Contexts(NamedTuple): + root: LiveMapBatchContext + profile: LiveMapBatchContext + score: LiveCounterBatchContext + name: PrimitiveBatchContext + + +# A call of each context method that reads the objects, and of each that writes +_READS = { + 'get': lambda c: c.root.get('name'), + 'compact_json': lambda c: c.root.compact_json(), + 'entries': lambda c: c.root.entries(), + 'keys': lambda c: c.root.keys(), + 'values': lambda c: c.root.values(), + 'size': lambda c: c.root.size(), + 'child-get': lambda c: c.profile.get('email'), + 'counter-value': lambda c: c.score.value(), + 'primitive-value': lambda c: c.name.value(), +} +_WRITES = { + 'set': lambda c: c.root.set('name', 'Bob'), + 'set-value-type': lambda c: c.root.set('team', LiveMap.create()), + 'remove': lambda c: c.root.remove('name'), + 'child-set': lambda c: c.profile.set('email', 'bob@example.com'), + 'increment': lambda c: c.score.increment(1), + 'decrement': lambda c: c.score.decrement(1), +} +# The calls that check whether the batch is open, but no channel preconditions +_VIEWS = { + 'id': lambda c: c.root.id, + 'primitive-id': lambda c: c.name.id, + 'as_live_map': lambda c: c.root.as_live_map(), + 'as_live_counter': lambda c: c.score.as_live_counter(), + 'as_primitive': lambda c: c.name.as_primitive(), +} + + +def _contexts(ctx): + return _Contexts(ctx, ctx.get('profile'), ctx.get('score'), ctx.get('name')) + + +@pytest.mark.parametrize('call', [*_READS.values(), *_WRITES.values()], ids=[*_READS, *_WRITES]) +async def test_rtbc4b_rtbc15b_contexts_check_preconditions_on_each_call(call): + """RTBC4b-RTBC15b: the contexts check the channel preconditions on every read (RTO25) and + write (RTO26), so once the channel detaches inside the block each of them raises 90001, and + nothing is published.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + contexts = _contexts(ctx) + await channel.detach() + + with pytest.raises(AblyException) as excinfo: + call(contexts) + _assert_error(excinfo, 90001) + + assert published == [] + + +@pytest.mark.parametrize('call', [*_VIEWS.values(), *_READS.values(), *_WRITES.values()], + ids=[*_VIEWS, *_READS, *_WRITES]) +async def test_rtbc16e_contexts_raise_once_the_batch_has_closed(call): + """RTBC16e, RTBC3b-RTBC15c: once its block has exited, every method of a batch's contexts, + those opened inside the block included, raises 40000.""" + (client, channel, root, mock_ws), published = await _synced() + + async with root.batch() as ctx: + contexts = _contexts(ctx) + + with pytest.raises(AblyException) as excinfo: + call(contexts) + _assert_error(excinfo, 40000) diff --git a/test/unit/liveobjects/conftest.py b/test/unit/liveobjects/conftest.py new file mode 100644 index 00000000..6b7cf0d3 --- /dev/null +++ b/test/unit/liveobjects/conftest.py @@ -0,0 +1,10 @@ +import pytest + +from test.uts.helpers.client import close_open_clients + + +@pytest.fixture(autouse=True) +async def close_clients(): + """Closes the clients a test built through the UTS helpers, whether or not its assertions held.""" + yield + await close_open_clients() diff --git a/test/unit/liveobjects/realtime_object_test.py b/test/unit/liveobjects/realtime_object_test.py new file mode 100644 index 00000000..c93786fb --- /dev/null +++ b/test/unit/liveobjects/realtime_object_test.py @@ -0,0 +1,365 @@ +"""A channel's `RealtimeObject`: waits for SYNCED the channel can no longer satisfy, object +messages that fail to decode or to apply, and the lifetime of the GC timer. + +Spec points: RTL33b1, RTO5, RTO5c, RTO8, RTO9, RTO10, RTO10c, RTO18, RTO19, RTO20e1, RTO23c1, +RTO27a. No UTS specification covers these cases, so they are written against the features +specification. + +Several cases race a channel state change against a coroutine resuming. The mock websocket +answers one client message with two frames, which the transport handles one after the other +before the coroutine awaiting the first of them resumes, as when the server's frames arrive in +one read. +""" + +import asyncio +import logging + +import pytest + +from ably.pubsub.liveobjects.defaults import GC_INTERVAL_MS +from ably.pubsub.liveobjects.enums import ObjectsSyncState +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.realtimeobject import RealtimeObject +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import await_connection_state, poll_until +from test.uts.helpers.clock import FakeClock, settle +from test.uts.helpers.mock_websocket import MockWebSocket, channel_error_message +from test.uts.objects.helpers.standard_test_pool import ( + ATTACH, + HAS_OBJECTS, + LWW, + STANDARD_POOL_OBJECTS, + build_ack_message, + build_counter_inc, + build_map_set, + build_object_message, + build_object_state, + objects_attached_message, + objects_channel_options, + objects_client, + objects_connected_message, + remote_serial, + setup_synced_channel, + standard_mock_websocket, +) + +# How long a test waits on an operation it expects to complete or fail, so that one which +# never settles fails with a timeout rather than running into the suite's own +OPERATION_TIMEOUT = 5 + +SCORE = 'counter:score@1000' + + +def _score(channel): + return channel.object._objects_pool[SCORE].data + + +def _assert_sync_wait_failure(excinfo, state_text, cause_code=None): + """Asserts an RTO23c1/RTO20e1 failure: 92008, status 400, naming why, with the channel's error.""" + assert excinfo.value.code == 92008 + assert excinfo.value.status_code == 400 + assert state_text in excinfo.value.message + if cause_code is not None: + assert excinfo.value.cause.code == cause_code + + +async def _restart_sync(channel, mock_ws): + """Sends an ATTACHED while attached, which restarts the sync (RTO4) with no OBJECT_SYNC to follow.""" + mock_ws.send_to_client(objects_attached_message('test', 'sync2:cursor', HAS_OBJECTS)) + await poll_until(lambda: channel.object._sync_state == ObjectsSyncState.SYNCING, + description='the sync to restart') + + +def _ack_then_channel_error(mock_ws_ref): + """An `on_object` handler answering each OBJECT with its ACK and, in the same read, a channel ERROR.""" + def on_object(message): + mock_ws_ref[0].send_to_client(build_ack_message(message['msgSerial'], ['t:9:0'])) + mock_ws_ref[0].send_to_client(channel_error_message('test', 90000, 'Channel error', 400)) + return on_object + + +# --- B1: a wait for SYNCED on a channel that has already left ATTACHED ---------------------- + +async def test_rto23c1_get_fails_when_the_channel_fails_with_its_attached(): + """RTO23c1: an ATTACHED and a channel ERROR read together leave the channel FAILED before + `get()` resumes from its attach, so `get()` begins its wait on a channel that will never + sync. It fails at once with 92008, the channel's error as the cause, rather than waiting + for a transition that has already happened.""" + mock_ws = None + + def on_message_from_client(message): + if message.get('action') == ATTACH: + channel_name = message.get('channel') + mock_ws.send_to_client(objects_attached_message(channel_name, 'sync1:cursor', HAS_OBJECTS)) + mock_ws.send_to_client(channel_error_message(channel_name, 90000, 'Channel error', 400)) + + mock_ws = MockWebSocket( + on_connection_attempt=lambda conn: conn.respond_with_success(objects_connected_message()), + on_message_from_client=on_message_from_client, + ) + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options()) + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(channel.object.get(), OPERATION_TIMEOUT) + + _assert_sync_wait_failure(excinfo, 'failed', cause_code=90000) + assert channel.state == ChannelState.FAILED + assert channel.object._sync_waiters == {} + + +async def test_rto20e1_publish_and_apply_fails_when_the_channel_fails_with_the_ack(): + """RTO20e1: an ACK and a channel ERROR read together leave the channel FAILED before the + write resumes, so its wait for SYNCED begins on a channel that will never sync. It fails at + once with 92008, the channel's error as the cause.""" + mock_ws_ref = [] + mock_ws = standard_mock_websocket(auto_ack=False, on_object=_ack_then_channel_error(mock_ws_ref)) + mock_ws_ref.append(mock_ws) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + await _restart_sync(channel, mock_ws) + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(root.set('name', 'Bob'), OPERATION_TIMEOUT) + + _assert_sync_wait_failure(excinfo, 'failed', cause_code=90000) + assert channel.state == ChannelState.FAILED + assert channel.object._sync_waiters == {} + + +async def test_rto20e1_publish_and_apply_fails_when_the_channel_detached_before_the_ack(): + """RTO20e1: a write whose channel detaches while its publish awaits the ACK begins its wait + for SYNCED on a DETACHED channel, and fails at once with 92008.""" + published = [] + mock_ws = standard_mock_websocket(auto_ack=False, on_object=published.append) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + await _restart_sync(channel, mock_ws) + + task = asyncio.ensure_future(root.get('score').as_live_counter().increment(10)) + await poll_until(lambda: len(published) == 1, description='the OBJECT to be published') + await channel.detach() + mock_ws.send_to_client(build_ack_message(published[0]['msgSerial'], ['t:9:0'])) + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + _assert_sync_wait_failure(excinfo, 'detached') + assert channel.object._sync_waiters == {} + + +async def test_rto27a_publish_and_apply_does_not_apply_once_the_channel_has_failed(): + """RTO27a, RTO20f: a write on a SYNCED channel whose ACK is read together with a channel + ERROR completes, as it was published, but is not applied locally: the channel's FAILED + has cleared the objects' data, and applying it would rebuild that data and notify + subscribers on a failed channel.""" + mock_ws_ref = [] + mock_ws = standard_mock_websocket(auto_ack=False, on_object=_ack_then_channel_error(mock_ws_ref)) + mock_ws_ref.append(mock_ws) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + calls = [] + root.get('score').subscribe(calls.append) + + await asyncio.wait_for(root.get('score').as_live_counter().increment(10), OPERATION_TIMEOUT) + await settle() + + assert channel.state == ChannelState.FAILED + assert _score(channel) == 0 + assert calls == [] + + +async def test_rtl33b1_get_fails_when_the_client_closes_during_the_attach(): + """RTL33b1, RTO23e: closing the client while `get()` attaches moves the ATTACHING channel to + DETACHED, which ends the attach without attaching. `get()` fails with the attach's error, + rather than waiting for a sync on a channel that has detached.""" + mock_ws = MockWebSocket( + on_connection_attempt=lambda conn: conn.respond_with_success(objects_connected_message()), + on_message_from_client=lambda message: None, # never answers the ATTACH + ) + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options()) + await await_connection_state(client, ConnectionState.CONNECTED) + + task = asyncio.ensure_future(channel.object.get()) + await poll_until(lambda: channel.state == ChannelState.ATTACHING, description='the channel to attach') + await client.close() + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert excinfo.value.code == 90001 + assert excinfo.value.status_code == 400 + assert channel.state == ChannelState.DETACHED + assert channel.object._sync_waiters == {} + + +# --- R1: object messages that fail to decode or to apply ------------------------------------ + +async def test_rto5_an_object_sync_message_that_fails_to_decode_is_skipped(caplog): + """RTO5, RTO5a4: one undecodable value in the last OBJECT_SYNC of a sequence costs only the + ObjectMessage carrying it. The rest of the ProtocolMessage, its cursor included, is + handled, so the sync completes and `get()` returns.""" + bad = build_object_state('map:bad@1000', {'aaa': 't:0'}, map={ + 'semantics': LWW, 'entries': {'k': {'data': {'json': '{not json'}, 'timeserial': 't:0'}}}) + mock_ws = standard_mock_websocket(sync_objects=(*STANDARD_POOL_OBJECTS, bad)) + + with caplog.at_level(logging.ERROR, logger='ably.pubsub.liveobjects'): + client, channel, root, mock_ws = await asyncio.wait_for( + setup_synced_channel('test', mock_ws=mock_ws), OPERATION_TIMEOUT) + + assert channel.object._sync_state == ObjectsSyncState.SYNCED + assert root.get('score').as_live_counter().value() == 100 + assert 'map:bad@1000' not in channel.object._objects_pool + assert any('failed to decode' in record.getMessage() for record in caplog.records) + + +async def test_rto8_an_object_message_that_fails_to_decode_costs_only_itself(caplog): + """RTO8: an OBJECT carrying an undecodable value and a valid operation applies the valid + one.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + + with caplog.at_level(logging.ERROR, logger='ably.pubsub.liveobjects'): + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'data', {'json': '{not json'}, remote_serial(0), 'remote'), + build_counter_inc(SCORE, 1, remote_serial(1), 'remote'), + ])) + await poll_until(lambda: _score(channel) == 101, description='the valid increment to apply') + + assert root.get('data').as_primitive().value() == {'tags': ['a', 'b']} + assert any('failed to decode' in record.getMessage() for record in caplog.records) + + +async def test_rto8_an_increment_by_a_non_number_fails_to_decode_and_costs_only_itself(caplog): + """RTO8, OD4c3: an increment by a string fails to decode rather than reaching the counter, + and the operations after it in the OBJECT still apply.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + + with caplog.at_level(logging.ERROR, logger='ably.pubsub.liveobjects'): + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc(SCORE, 'one', remote_serial(0), 'remote'), + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(1), 'remote'), + ])) + await poll_until(lambda: root.get('name').as_primitive().value() == 'Bob', + description='the MAP_SET after the undecodable increment to apply') + + assert _score(channel) == 100 + assert any('failed to decode' in record.getMessage() for record in caplog.records) + + +async def test_rto9_an_object_message_that_fails_to_apply_costs_only_itself(caplog, monkeypatch): + """RTO9: an operation that decodes but raises as it is applied is logged and skipped, and + the operations after it in the OBJECT still apply.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + + def apply_counter_inc(self, *args, **kwargs): + raise RuntimeError('apply_counter_inc failed') + monkeypatch.setattr(InternalLiveCounter, 'apply_counter_inc', apply_counter_inc) + + with caplog.at_level(logging.ERROR, logger='ably.pubsub.liveobjects'): + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc(SCORE, 1, remote_serial(0), 'remote'), + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(1), 'remote'), + ])) + await poll_until(lambda: root.get('name').as_primitive().value() == 'Bob', + description='the MAP_SET after the failing increment to apply') + + assert _score(channel) == 100 + assert any('could not be applied' in record.getMessage() for record in caplog.records) + + +async def test_rto5c_an_object_state_that_fails_to_apply_does_not_stall_the_sync(caplog, monkeypatch): + """RTO5c: an object state that decodes but raises as it is applied is logged and skipped, + and the sync still completes.""" + bad = build_object_state('counter:bad@1000', {'aaa': 't:0'}, counter={'count': 1}) + mock_ws = standard_mock_websocket(sync_objects=(*STANDARD_POOL_OBJECTS, bad)) + + replace_data = InternalLiveCounter.replace_data + + def failing_replace_data(self, object_message): + if self.object_id == 'counter:bad@1000': + raise RuntimeError('replace_data failed') + return replace_data(self, object_message) + monkeypatch.setattr(InternalLiveCounter, 'replace_data', failing_replace_data) + + with caplog.at_level(logging.ERROR, logger='ably.pubsub.liveobjects'): + client, channel, root, mock_ws = await asyncio.wait_for( + setup_synced_channel('test', mock_ws=mock_ws), OPERATION_TIMEOUT) + + assert channel.object._sync_state == ObjectsSyncState.SYNCED + assert root.get('score').as_live_counter().value() == 100 + assert any('could not be applied' in record.getMessage() for record in caplog.records) + + +# --- R2: the GC timer and waits of a released channel --------------------------------------- + +async def test_rto10_a_released_channel_stops_its_gc_timer(): + """RTO10a: the GC timer every ATTACHED schedules stops when its channel is detached, and + once the channel is released (RTS4) nothing schedules it again.""" + clock = FakeClock() + client, channel, root, mock_ws = await setup_synced_channel('test', clock=clock) + timer = channel.object._gc_timer + assert timer is not None + + await channel.detach() + client.channels.release('test') + channel.object._schedule_gc_timer() + await client.close() + await clock.advance(GC_INTERVAL_MS) + + assert timer.cancelled + assert channel.object._gc_timer is None + + +async def test_rto23c1_a_wait_for_synced_fails_when_its_channel_is_released(): + """RTO23c1: a released channel receives nothing more, so it can never sync. A wait for + SYNCED begun after the channel is released fails with 92008 at once.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + await channel.detach() + client.channels.release('test') + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(channel.object._wait_for_synced('The object could not be retrieved'), + OPERATION_TIMEOUT) + _assert_sync_wait_failure(excinfo, 'released') + + +# --- R4: a GC sweep that raises ------------------------------------------------------------- + +async def test_rto10c_a_gc_sweep_that_raises_is_logged_and_the_next_still_runs(caplog, monkeypatch): + """RTO10c: a sweep that raises is logged, and the next is scheduled all the same; the error + does not escape the timer's callback.""" + clock = FakeClock() + client, channel, root, mock_ws = await setup_synced_channel('test', clock=clock) + sweeps = [] + + def collect_garbage(grace_period_ms, now_ms): + sweeps.append(now_ms) + if len(sweeps) == 1: + raise RuntimeError('the sweep failed') + + monkeypatch.setattr(channel.object._objects_pool, 'collect_garbage', collect_garbage) + + with caplog.at_level(logging.ERROR, logger='ably.pubsub.liveobjects'): + await clock.advance(GC_INTERVAL_MS) + assert len(sweeps) == 1 + assert any('GC sweep raised' in record.getMessage() for record in caplog.records) + assert channel.object._gc_timer is not None + + await clock.advance(GC_INTERVAL_MS) + assert len(sweeps) == 2 + + +# --- on and off -------------------------------------------------------------------------- + +@pytest.mark.parametrize('register', [ + lambda realtime_object: realtime_object.on('synched', lambda: None), + lambda realtime_object: realtime_object.off('synched', lambda: None), +]) +def test_rto18_rto19_an_unknown_event_raises_40003(register): + """RTO18, RTO19: `on` and `off` take an `ObjectsEvent`; any other event is an invalid + argument, raised as AblyException 40003 like the other argument errors.""" + with pytest.raises(AblyException) as excinfo: + register(RealtimeObject()) + + assert excinfo.value.code == 40003 + assert excinfo.value.status_code == 400 diff --git a/test/unit/liveobjects/subscription_test.py b/test/unit/liveobjects/subscription_test.py new file mode 100644 index 00000000..17fd56ab --- /dev/null +++ b/test/unit/liveobjects/subscription_test.py @@ -0,0 +1,69 @@ +"""Which listeners a dispatch calls when listeners are added or removed during it. + +Spec points: RTLO4b4c3a (`subscribe` listeners on a live object) and RTO24b (path +subscriptions). The specification does not say how a dispatch treats registrations changed +by its own listeners; the library takes one snapshot per dispatch, so that a listener added +during it is not called by it, and skips a listener removed during it, for both kinds. +""" + +from test.uts.helpers.clock import settle +from test.uts.objects.helpers.standard_test_pool import ( + build_counter_inc, + build_map_set, + build_object_message, + remote_serial, + setup_synced_channel, +) + +SCORE = 'counter:score@1000' + + +async def test_rtlo4b4c3a_a_listener_unsubscribed_during_a_dispatch_is_not_called(): + """RTLO4b4c3a: a listener that another listener unsubscribes earlier in the same dispatch + is not called for that update.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + counter = root.get('score').instance().as_live_counter() + calls = [] + second = None + + def first(event): + calls.append('first') + second.unsubscribe() + + counter.subscribe(first) + second = counter.subscribe(lambda event: calls.append('second')) + await counter.increment(1) + await settle() + + assert calls == ['first'] + + +async def test_rto24b_a_subscription_made_during_a_dispatch_is_not_called_by_it(): + """RTO24b: a path subscription made by a listener is not called by the dispatch that made + it, even for a later path to the same object, and is called by the next update.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + # root.alias references the same counter as root.score, so the counter has two paths + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'alias', {'objectId': SCORE}, remote_serial(0), 'remote')])) + await settle() + assert root.get('alias').as_live_counter().value() == 100 + + late_calls = [] + added = [] + + def first(event): + if not added: + added.append(root.subscribe(lambda late_event: late_calls.append(late_event.object.path()))) + + root.get('score').subscribe(first) + root.get('alias').subscribe(first) + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc(SCORE, 1, remote_serial(1), 'remote')])) + await settle() + assert added + assert late_calls == [] + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc(SCORE, 1, remote_serial(2), 'remote')])) + await settle() + assert sorted(late_calls) == ['alias', 'score'] diff --git a/test/unit/liveobjects/typed_views_test.py b/test/unit/liveobjects/typed_views_test.py new file mode 100644 index 00000000..6f3ccb49 --- /dev/null +++ b/test/unit/liveobjects/typed_views_test.py @@ -0,0 +1,422 @@ +"""The typed views: `type()` and `exists()`, the unchecked path views, the checked instance views +and the snapshots `compact_json` takes through them. + +Spec points: RTTS2-RTTS10 (ably/specification#491), as LODR-061 binds them: one +`PrimitivePathObject` and `PrimitiveInstance` whose `value(expected)` stands for the six +per-primitive classes (RTTS6h), and `ValueType.UNKNOWN` for a value in none of the known +categories. No UTS specification covers the typed views, so these tests are written against +the features specification and LODR-061. + +Each test drives the standard synced pool over the mock websocket. +""" + +import json +import typing + +import pytest + +from ably.pubsub.liveobjects.liveobject import LiveObject, LiveObjectUpdate +from ably.pubsub.server.liveobjects import ( + Instance, + LiveCounterInstance, + LiveCounterPathObject, + LiveMapInstance, + LiveMapPathObject, + PathObject, + PrimitiveInstance, + PrimitivePathObject, + ValueType, +) +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import poll_until +from test.uts.objects.helpers.standard_test_pool import ( + build_map_set, + build_object_message, + remote_serial, + setup_synced_channel, +) + +# The type of each value in the standard pool, and of the JSON array each test adds to it +POOL_VALUE_TYPES = { + '': ValueType.LIVE_MAP, + 'name': ValueType.STRING, + 'age': ValueType.NUMBER, + 'active': ValueType.BOOLEAN, + 'avatar': ValueType.BINARY, + 'data': ValueType.JSON_OBJECT, + 'tags': ValueType.JSON_ARRAY, + 'score': ValueType.LIVE_COUNTER, + 'profile': ValueType.LIVE_MAP, + 'profile.prefs.theme': ValueType.STRING, +} + + +async def _synced_root(): + """The standard synced root, with a JSON array added at `tags`.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + await root.set('tags', ['a', 'b']) + return root + + +def _at(root, path): + """The path object at the dotted `path` from `root`, `root` itself for the empty path.""" + return root.at(path) if path else root + + +def _assert_error(excinfo, code): + assert excinfo.value.code == code + assert excinfo.value.status_code == 400 + + +class _FutureLiveObject(LiveObject): + """A live object of a type this SDK does not know, standing in for one a later protocol adds.""" + + def apply_operation(self, object_message, source): + return False + + def replace_data(self, object_message): + return LiveObjectUpdate(noop=True) + + def clear_data(self): + pass + + @staticmethod + def diff(previous_data, new_data, *, for_tombstone=False): + return LiveObjectUpdate(noop=True) + + +# --- type() and exists() ------------------------------------------------------------------ + +@pytest.mark.parametrize('path, value_type', POOL_VALUE_TYPES.items(), ids=[p or 'root' for p in POOL_VALUE_TYPES]) +async def test_rtts4b_type_is_the_type_of_the_resolved_value(path, value_type): + """RTTS4a, RTTS4b, RTTS2a1-RTTS2a8: `type()` is the `ValueType` of the value the path resolves to, + and `exists()` is True for it.""" + root = await _synced_root() + + assert _at(root, path).type() is value_type + assert _at(root, path).exists() is True + + +@pytest.mark.parametrize('path', ['missing', 'name.first', 'score.value', 'data.tags', 'profile.prefs.missing']) +async def test_rtts4b_type_is_none_where_nothing_resolves(path): + """RTTS4a3, RTTS4b3: `type()` is None and `exists()` False for a path that does not resolve, + including one that continues below a primitive or a counter.""" + root = await _synced_root() + + assert root.at(path).type() is None + assert root.at(path).exists() is False + + +async def test_rtts4b_type_follows_the_value_at_the_path(): + """RTTS4b, RTTS8a: a path's type is read when it is asked for, so it follows a change to the + value at the path; an instance's type is fixed by the value it wraps.""" + root = await _synced_root() + score = root.get('score') + instance = score.instance() + + await root.set('score', 'retired') + assert score.type() is ValueType.STRING + assert instance.type is ValueType.LIVE_COUNTER + + await root.remove('score') + assert score.type() is None + assert score.exists() is False + + +async def test_rtts2a9_type_is_unknown_for_a_value_in_no_known_category(): + """RTTS2a9: a value that resolves but falls into none of the known categories is + `ValueType.UNKNOWN`, which is distinct from None for no value, and the typed views read + None from it.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + channel.object._objects_pool['future:thing@1000'] = _FutureLiveObject('future:thing@1000') + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'thing', {'objectId': 'future:thing@1000'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: 'thing' in root.keys(), description='the reference to be applied') + + thing = root.get('thing') + assert thing.type() is ValueType.UNKNOWN + assert thing.exists() is True + assert thing.as_primitive().value() is None + assert thing.as_live_counter().value() is None + assert thing.as_live_map().size() is None + assert thing.as_live_map().keys() == [] + assert thing.instance().type is ValueType.UNKNOWN + + +async def test_rtts4a1_rtts4b1_type_and_exists_check_access_preconditions(): + """RTTS4a1, RTTS4b1, RTO25b: `type()` and `exists()` raise 90001 on a detached channel.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + + await channel.detach() + assert channel.state == ChannelState.DETACHED + + for read in (root.get('name').type, root.get('name').exists): + with pytest.raises(AblyException) as excinfo: + read() + _assert_error(excinfo, 90001) + + +# --- The unchecked path views ------------------------------------------------------------- + +async def test_rtts5_path_views_rewrap_the_same_path(): + """RTTS5, RTTS6d: `channel.object.get()` is a `LiveMapPathObject`; navigation gives base + `PathObject`s, and each view helper gives the typed view of the same path, without reading + anything.""" + root = await _synced_root() + assert isinstance(root, LiveMapPathObject) + + path = root.get('profile').get('prefs') + assert type(path) is PathObject + for view, view_type in ((path.as_live_map(), LiveMapPathObject), + (path.as_live_counter(), LiveCounterPathObject), + (path.as_primitive(), PrimitivePathObject)): + assert type(view) is view_type + assert view.path() == 'profile.prefs' + + assert path.as_live_map().get('theme').as_primitive().value() == 'dark' + # A view of a path that does not resolve is still a view + assert root.get('missing').as_live_counter().path() == 'missing' + + +async def test_rtts5d1_mismatched_path_views_read_none(): + """RTTS5d, RTTS5d1: a path view never raises for the type at its path; a read through a view + that does not match it gives None, or [] for the collections.""" + root = await _synced_root() + + name_as_map = root.get('name').as_live_map() + assert name_as_map.keys() == [] + assert name_as_map.entries() == [] + assert name_as_map.values() == [] + assert name_as_map.size() is None + + assert root.get('name').as_live_counter().value() is None + assert root.get('score').as_primitive().value() is None + assert root.get('score').as_live_map().size() is None + assert root.as_primitive().value() is None + assert root.as_live_counter().value() is None + assert root.get('missing').as_primitive().value() is None + + +async def test_rtts5d2_writes_through_mismatched_path_views_raise(): + """RTTS5d2: a write through a path view raises 92007 when the path resolves to another type, + and 92005 when it does not resolve.""" + root = await _synced_root() + + for write, code in ((lambda: root.get('name').as_live_map().set('k', 'v'), 92007), + (lambda: root.get('profile').as_live_counter().increment(), 92007), + (lambda: root.get('score').as_live_map().remove('k'), 92007), + (lambda: root.get('missing').as_live_counter().decrement(), 92005), + (lambda: root.at('name.first').as_live_map().set('k', 'v'), 92005)): + with pytest.raises(AblyException) as excinfo: + await write() + _assert_error(excinfo, code) + + assert root.get('name').as_primitive().value() == 'Alice' + + +async def test_rtts6b_counter_view_value_is_only_a_counters(): + """RTTS6b: `LiveCounterPathObject.value()` is the counter's value as a float, and None for + anything else, a number included.""" + root = await _synced_root() + + score = root.get('score').as_live_counter().value() + assert score == 100 + assert type(score) is float + assert root.get('age').as_live_counter().value() is None + assert root.as_live_counter().value() is None + + +@pytest.mark.parametrize('path, expected, value', [ + ('name', str, 'Alice'), + ('name', float, None), + ('age', float, 30.0), + ('age', str, None), + ('age', bool, None), + ('active', bool, True), + ('active', float, None), + ('avatar', bytes, b'\x01\x02\x03'), + ('avatar', str, None), + ('data', dict, {'tags': ['a', 'b']}), + ('data', list, None), + ('tags', list, ['a', 'b']), + ('tags', dict, None), + ('score', float, None), + ('profile', dict, None), + ('missing', str, None), +], ids=lambda param: param.__name__ if isinstance(param, type) else None) +async def test_rtts6c_value_expected_filters_by_type(path, expected, value): + """RTTS6c, RTTS6h: `value(expected)` is the primitive at the path only if it is of the expected + type, judged by its wire type, so a boolean is never a number and a counter or map is never + a primitive.""" + root = await _synced_root() + + result = root.get(path).as_primitive().value(expected) + assert result == value + assert type(result) is type(value) + + +async def test_rtts6c_value_without_expected_is_any_primitive(): + """RTTS6c, RTTS6h: `value()` with no `expected` is the primitive at the path whatever its type, + and None for a live object.""" + root = await _synced_root() + + assert {path: _at(root, path).as_primitive().value() for path in POOL_VALUE_TYPES} == { + '': None, + 'name': 'Alice', + 'age': 30.0, + 'active': True, + 'avatar': b'\x01\x02\x03', + 'data': {'tags': ['a', 'b']}, + 'tags': ['a', 'b'], + 'score': None, + 'profile': None, + 'profile.prefs.theme': 'dark', + } + + +async def test_rtts6c_value_rejects_an_unsupported_expected_type(): + """RTTS6c: `expected` is one of str, float, bool, bytes, list and dict; anything else, a + subclass of one of them included, is a TypeError, raised before the preconditions are + checked.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + age = root.get('age').as_primitive() + name = root.get('name').as_primitive() + + for expected in (int, object, tuple, 'str', typing.List, type('Text', (str,), {})): + with pytest.raises(TypeError): + name.value(expected) + + await channel.detach() + with pytest.raises(TypeError): + age.value(int) + with pytest.raises(AblyException) as excinfo: + age.value(float) + _assert_error(excinfo, 90001) + + +async def test_rtts6c_value_is_a_copy_of_a_json_value(): + """RTTS6c: a JSON value read through a view is the reader's own, so changing it leaves the + object it was read from unchanged.""" + root = await _synced_root() + + data = root.get('data').as_primitive().value(dict) + data['tags'].append('c') + + assert root.get('data').as_primitive().value(dict) == {'tags': ['a', 'b']} + + +# --- The checked instance views ----------------------------------------------------------- + +@pytest.mark.parametrize('path, value_type', POOL_VALUE_TYPES.items(), ids=[p or 'root' for p in POOL_VALUE_TYPES]) +async def test_rtts8a_instance_type_is_the_type_of_the_wrapped_value(path, value_type): + """RTTS7e, RTTS8a: every instance is the typed subclass matching the value it wraps, and its + `type` property is that value's `ValueType`.""" + root = await _synced_root() + + instance = _at(root, path).instance() + + assert instance.type is value_type + expected_class = {ValueType.LIVE_MAP: LiveMapInstance, ValueType.LIVE_COUNTER: LiveCounterInstance}.get( + value_type, PrimitiveInstance) + assert type(instance) is expected_class + + +async def test_rtts9d_instance_views_are_checked(): + """RTTS9, RTTS9d: an instance's view helper returns the instance for the type it wraps and + raises 92007 for any other, where the same helper on a path never raises.""" + root = await _synced_root() + instances = { + 'profile': (root.get('profile').instance(), 'as_live_map'), + 'score': (root.get('score').instance(), 'as_live_counter'), + 'name': (root.get('name').instance(), 'as_primitive'), + } + + for path, (instance, matching_helper) in instances.items(): + assert getattr(instance, matching_helper)() is instance + for helper in {'as_live_map', 'as_live_counter', 'as_primitive'} - {matching_helper}: + with pytest.raises(AblyException) as excinfo: + getattr(instance, helper)() + _assert_error(excinfo, 92007) + # The same request on the path is an unchecked expectation + assert getattr(root.get(path), helper)().path() == path + + +async def test_rtts10_instance_id_is_a_property(): + """RTTS10a, RTTS10b, RTINS3: `id` is a property: the object id of a map or counter instance, + and None for a primitive one.""" + root = await _synced_root() + + assert root.instance().id == 'root' + assert root.get('profile').instance().id == 'map:profile@1000' + assert root.get('score').instance().id == 'counter:score@1000' + assert root.get('name').instance().id is None + + +async def test_rtts7a_instance_compact_json_is_of_the_wrapped_type(): + """RTINS11, RTTS7a1, RTTS7a3: a map instance's snapshot is a JSON object, a counter + instance's is its value as a float, and a primitive instance's is the primitive.""" + root = await _synced_root() + + profile = root.get('profile').instance().compact_json() + score = root.get('score').instance().compact_json() + + assert profile == {'email': 'alice@example.com', 'nested_counter': 5, 'prefs': {'theme': 'dark'}} + assert type(score) is float + assert score == 100 + assert root.get('name').instance().compact_json() == 'Alice' + assert root.get('avatar').instance().compact_json() == 'AQID' + + +async def test_rtpo14b_compact_json_is_json_serializable(): + """RTPO14b: binary is a base64 string, and a map reached a second time in one snapshot is + its object id, through a second reference as through a cycle, so `json.dumps` takes the + snapshot.""" + client, channel, root, mock_ws = await setup_synced_channel('test') + # root.alias references the same map as root.profile + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'alias', {'objectId': 'map:profile@1000'}, remote_serial(0), 'remote')])) + await poll_until(lambda: 'alias' in root.keys(), description='the reference to be applied') + + result = root.compact_json() + + assert result['avatar'] == 'AQID' + assert result['profile'] == {'email': 'alice@example.com', 'nested_counter': 5, 'prefs': {'theme': 'dark'}} + assert result['alias'] == {'objectId': 'map:profile@1000'} + assert json.loads(json.dumps(result)) == result + + +@pytest.mark.parametrize('typed_class, present, absent', [ + (PathObject, + {'path', 'type', 'exists', 'get', 'at', 'instance', 'compact_json', 'subscribe', + 'as_live_map', 'as_live_counter', 'as_primitive'}, + {'compact', 'value', 'entries', 'keys', 'values', 'size', 'set', 'remove', 'increment', 'decrement', + 'batch'}), + (LiveMapPathObject, + {'entries', 'keys', 'values', 'size', 'set', 'remove', 'batch'}, + {'value', 'increment', 'decrement'}), + (LiveCounterPathObject, + {'value', 'increment', 'decrement', 'batch'}, + {'entries', 'keys', 'values', 'size', 'set', 'remove'}), + (PrimitivePathObject, + {'value'}, + {'entries', 'keys', 'values', 'size', 'set', 'remove', 'increment', 'decrement', 'batch'}), + (Instance, + {'id', 'type', 'get', 'compact_json', 'as_live_map', 'as_live_counter', 'as_primitive'}, + {'compact', 'value', 'entries', 'size', 'set', 'remove', 'increment', 'decrement', 'subscribe', 'batch'}), + (LiveMapInstance, + {'entries', 'keys', 'values', 'size', 'set', 'remove', 'subscribe', 'batch'}, + {'value', 'increment', 'decrement'}), + (LiveCounterInstance, + {'value', 'increment', 'decrement', 'subscribe', 'batch'}, + {'entries', 'keys', 'values', 'size', 'set', 'remove'}), + (PrimitiveInstance, + {'value'}, + {'entries', 'keys', 'values', 'size', 'set', 'remove', 'increment', 'decrement', 'subscribe', 'batch'}), +], ids=lambda param: param.__name__ if isinstance(param, type) else '') +def test_rtts3_rtts10_methods_are_partitioned_by_type(typed_class, present, absent): + """RTTS3, RTTS6, RTTS7, RTTS10: each typed view carries the methods of its own type only, so + `subscribe` is on map and counter instances alone and only live objects can be batched.""" + assert {name for name in present if not hasattr(typed_class, name)} == set() + assert {name for name in absent if hasattr(typed_class, name)} == set() diff --git a/test/unit/liveobjects/value_types_test.py b/test/unit/liveobjects/value_types_test.py new file mode 100644 index 00000000..2d20f893 --- /dev/null +++ b/test/unit/liveobjects/value_types_test.py @@ -0,0 +1,43 @@ +"""The `LiveMap` and `LiveCounter` blueprints a write evaluates: what `create` captures, and when +their contents are validated. + +Spec points: RTLMV3d (a blueprint is immutable), RTLM20e1 and RTLMV4c (value validation), RTO16 +(the server time an object id is generated from). +""" + +import pytest + +from ably.pubsub.liveobjects.valuetypes import LiveCounter, LiveMap, evaluate +from ably.pubsub.util.exceptions import AblyException +from test.uts.objects.helpers.standard_test_pool import setup_synced_channel, time_mock_http + +SERVER_TIME_MS = 1_700_000_123_000 + + +def test_rtlmv3d_a_blueprint_does_not_change_with_the_values_it_was_created_from(): + """RTLMV3d: changing a JSON value or a `bytearray` after passing it to `LiveMap.create` does + not change the blueprint, at any depth.""" + meta = {'tags': ['a']} + raw = bytearray(b'\x01') + blueprint = LiveMap.create({'meta': meta, 'raw': raw}) + + meta['tags'].append('b') + meta['owner'] = 'Bob' + raw.append(2) + + map_create = evaluate(blueprint, SERVER_TIME_MS)[-1].operation.map_create_with_object_id.derived_from + assert map_create.entries['meta'].data.json == {'tags': ['a']} + assert map_create.entries['raw'].data.bytes == b'\x01' + + +async def test_rtlm20e1_an_invalid_blueprint_is_rejected_before_the_server_time_is_fetched(): + """RTLM20e1, RTLMV4c, RTO16: `set` validates a blueprint's contents before it asks the server + for the time its object ids are generated from, so an invalid one costs no request.""" + mock_http = time_mock_http() + client, channel, root, mock_ws = await setup_synced_channel('test', mock_http=mock_http) + + with pytest.raises(AblyException) as excinfo: + await root.set('team', LiveMap.create({'lead': 'Carol', 'points': LiveCounter.create('ten')})) + + assert excinfo.value.code == 40003 + assert [request for request in mock_http.captured_requests if request.path == '/time'] == [] diff --git a/test/unit/message_test.py b/test/unit/message_test.py index 917af30c..f49e9bf9 100644 --- a/test/unit/message_test.py +++ b/test/unit/message_test.py @@ -1,4 +1,4 @@ -import ably.types.message +import ably.pubsub.types.message # TM2a, TM2c, TM2f @@ -14,7 +14,7 @@ def test_update_inner_message_fields_tm2(): } ] } - ably.types.message.Message.update_inner_message_fields(proto_msg) + ably.pubsub.types.message.Message.update_inner_message_fields(proto_msg) messages: list[dict] = proto_msg.get('messages') msg_index = 0 for msg in messages: @@ -37,7 +37,7 @@ def test_update_inner_message_fields_for_presence_msg_tm2(): } ] } - ably.types.message.Message.update_inner_message_fields(proto_msg) + ably.pubsub.types.message.Message.update_inner_message_fields(proto_msg) presence_messages: list[dict] = proto_msg.get('presence') msg_index = 0 for presence_msg in presence_messages: @@ -49,7 +49,7 @@ def test_update_inner_message_fields_for_presence_msg_tm2(): # RSL6b def test_invalid_base64_data_is_delivered_with_remaining_encoding(): - message = ably.types.message.Message.from_encoded({ + message = ably.pubsub.types.message.Message.from_encoded({ 'name': 'event', 'data': 'encrypted-data-here', 'encoding': 'custom-encryption/base64', diff --git a/test/unit/mutable_message_test.py b/test/unit/mutable_message_test.py index 8ce603b9..e30ba443 100644 --- a/test/unit/mutable_message_test.py +++ b/test/unit/mutable_message_test.py @@ -1,5 +1,5 @@ -from ably import MessageAction, MessageOperation, MessageVersion, UpdateDeleteResult -from ably.types.message import Message +from ably.pubsub.server import MessageAction, MessageOperation, MessageVersion, UpdateDeleteResult +from ably.pubsub.types.message import Message def test_message_version_none_values_filtered(): @@ -100,7 +100,7 @@ def test_message_version_serialization(): def test_message_extras_preserved_in_as_dict(): """Test that extras are included when a Message with extras is serialized. - Regression test: _send_update() in both RestChannel and RealtimeChannel + Regression test: _send_update() in both HttpChannel and RealtimeChannel constructed a new Message without copying extras or annotations from the user-supplied message, violating RSL15b/RTL32b which require "whatever fields were in the user-supplied Message" to be sent. @@ -137,7 +137,7 @@ def test_message_extras_none_excluded_from_as_dict(): # RSL15b, RTL32b, TM2u def test_message_annotations_preserved_in_as_dict(): """Test that annotations are included when a Message with annotations is serialized.""" - from ably.types.message import MessageAnnotations + from ably.pubsub.types.message import MessageAnnotations annotations = MessageAnnotations(summary={'reaction:distinct.v1': {'thumbsup': 5}}) message = Message( name='test', diff --git a/test/unit/options_test.py b/test/unit/options_test.py index d3ba6129..84cc963b 100644 --- a/test/unit/options_test.py +++ b/test/unit/options_test.py @@ -1,7 +1,7 @@ import pytest -from ably.types.options import Options -from ably.util.exceptions import AblyException +from ably.pubsub.types.options import Options +from ably.pubsub.util.exceptions import AblyException # REC1b1: endpoint is incompatible with deprecated options diff --git a/test/unit/pubsub_server_test.py b/test/unit/pubsub_server_test.py new file mode 100644 index 00000000..0da6f813 --- /dev/null +++ b/test/unit/pubsub_server_test.py @@ -0,0 +1,182 @@ +import inspect +import typing + +import pytest + +import ably.pubsub.server as server +import ably.pubsub.server.liveobjects as server_liveobjects +import ably.pubsub.server.sync as server_sync +from ably.pubsub.http.http import DefaultPubSubHttpClient +from ably.pubsub.prototypes import PubSubHttpClient, PubSubRealtimeClient +from ably.pubsub.realtime.realtime import DefaultPubSubRealtimeClient +from ably.pubsub.server import create_http_client, create_realtime_client +from ably.pubsub.sync.http.http import DefaultPubSubHttpClientSync +from ably.pubsub.sync.prototypes import PubSubHttpClient as PubSubHttpClientSync + + +def protocol_members(protocol): + """The members `protocol` declares. + + Python 3.12 records these on the class. Earlier versions only have typing's + own helper, which is the function that builds that attribute, so the two + agree. Reading `vars(protocol)` instead would sweep in `__init__`, + `__abstractmethods__`, `__parameters__` and the rest of the Protocol + machinery, and report them as members the client fails to implement. + """ + members = getattr(protocol, '__protocol_attrs__', None) + if members is None: + members = typing._get_protocol_attrs(protocol) + return set(members) + + +def implementation_classes(protocol, cls): + """The classes in `cls`'s MRO that implement, rather than declare, the prototype. + + The clients subclass their prototype, so every prototype member is reachable + on them whether or not they implement it — an unimplemented one resolves to + the prototype's `...` stub and silently returns None. Dropping the + prototype's own MRO leaves only the classes that can genuinely define a + member. + """ + return [k for k in cls.__mro__ if k not in protocol.__mro__] + + +def declared_member(classes, name): + """The attribute `name` as the first class in `classes` defines it, if any.""" + for k in classes: + if name in k.__dict__: + return k.__dict__[name] + return None + + +def parameter_shape(member): + """A member's callable shape: whether it awaits, and its parameters. + + Annotations are left out deliberately — the prototype spells its types as + strings under `from __future__ import annotations` while the clients + evaluate theirs, and return types are a type checker's business. Parameter + names, kinds and which ones are required are what callers depend on. + """ + fn = member.fget if isinstance(member, property) else member + parameters = [ + (p.name, p.kind, p.default is inspect.Parameter.empty) + for p in inspect.signature(fn).parameters.values() + ] + return inspect.iscoroutinefunction(fn), parameters + + +class TestFactories: + def test_http_factory_returns_a_client(self): + client = create_http_client(token='foo') + assert isinstance(client, DefaultPubSubHttpClient) + + async def test_realtime_factory_returns_a_client(self): + client = create_realtime_client(key='foo:bar', auto_connect=False) + assert isinstance(client, DefaultPubSubRealtimeClient) + await client.close() + + def test_sync_http_factory_returns_a_client(self): + assert isinstance(server_sync.create_http_client(token='foo'), DefaultPubSubHttpClientSync) + + def test_factory_passes_options_through(self): + client = create_http_client(token='foo', client_id='me') + assert client.client_id == 'me' + + +class TestConstructorsAreInternal: + def test_rest_constructor_is_rejected(self): + with pytest.raises(TypeError) as excinfo: + DefaultPubSubHttpClient(token='foo') + assert 'ably.pubsub.server.create_http_client' in str(excinfo.value) + + def test_realtime_constructor_is_rejected(self): + with pytest.raises(TypeError) as excinfo: + DefaultPubSubRealtimeClient(key='foo:bar', auto_connect=False) + assert 'ably.pubsub.server.create_realtime_client' in str(excinfo.value) + + def test_sync_rest_constructor_is_rejected(self): + with pytest.raises(TypeError) as excinfo: + DefaultPubSubHttpClientSync(token='foo') + assert 'ably.pubsub.server.sync.create_http_client' in str(excinfo.value) + + def test_permission_does_not_leak_past_the_factory(self): + create_http_client(token='foo') + with pytest.raises(TypeError): + DefaultPubSubHttpClient(token='foo') + + @pytest.mark.parametrize('name', ['DefaultPubSubHttpClient', 'DefaultPubSubRealtimeClient']) + def test_client_classes_are_not_exported(self, name): + assert not hasattr(server, name) + assert name not in server.__all__ + + def test_sync_client_class_is_not_exported(self): + assert not hasattr(server_sync, 'DefaultPubSubHttpClientSync') + assert 'DefaultPubSubHttpClientSync' not in server_sync.__all__ + + +# Every prototype paired with the client declared to implement it, the +# synchronous flavour included — unasync generates both sides of that pair, +# so the pairing is worth checking rather than assuming. +PROTOTYPE_IMPLEMENTATIONS = [ + (PubSubHttpClient, DefaultPubSubHttpClient), + (PubSubRealtimeClient, DefaultPubSubRealtimeClient), + (PubSubHttpClientSync, DefaultPubSubHttpClientSync), +] + + +class TestPrototypes: + @pytest.mark.parametrize('protocol,cls', PROTOTYPE_IMPLEMENTATIONS) + def test_client_is_declared_to_satisfy_its_prototype(self, protocol, cls): + assert protocol in cls.__mro__ + + @pytest.mark.parametrize('protocol,cls', PROTOTYPE_IMPLEMENTATIONS) + def test_client_implements_every_prototype_member(self, protocol, cls): + owners = implementation_classes(protocol, cls) + missing = sorted( + member for member in protocol_members(protocol) + if declared_member(owners, member) is None + ) + assert missing == [] + + @pytest.mark.parametrize('protocol,cls', PROTOTYPE_IMPLEMENTATIONS) + def test_client_members_match_the_prototype_signature(self, protocol, cls): + owners = implementation_classes(protocol, cls) + mismatched = {} + for member in sorted(protocol_members(protocol)): + expected = parameter_shape(declared_member(protocol.__mro__, member)) + actual = parameter_shape(declared_member(owners, member)) + if expected != actual: + mismatched[member] = {'prototype': expected, 'client': actual} + assert mismatched == {} + + @pytest.mark.parametrize('name', ['PubSubHttpClient', 'PubSubRealtimeClient']) + def test_prototypes_are_exported(self, name): + assert name in server.__all__ + + def test_sync_prototype_is_exported(self): + assert 'PubSubHttpClient' in server_sync.__all__ + + +class TestPublicTypes: + # update_message() takes a Message, so callers need a public way to build one + def test_message_is_exported(self): + assert 'Message' in server.__all__ + assert 'Message' in server_sync.__all__ + + # RSE2: channel encryption needs a public way to generate a key and build cipher params + @pytest.mark.parametrize('name', ['generate_random_key', 'get_default_params']) + def test_crypto_helpers_are_exported(self, name): + assert name in server.__all__ + assert name in server_sync.__all__ + + # LiveObjects names live in their own module, named for the product (PDR-080), so + # that generic ones such as Batch and Subscription stay apart from the Pub/Sub names + @pytest.mark.parametrize('name', ['Batch', 'LiveCounter', 'LiveMap', 'PathObject', 'Subscription']) + def test_liveobjects_names_are_exported_from_their_own_module(self, name): + assert name in server_liveobjects.__all__ + assert name not in server.__all__ + assert not hasattr(server, name) + + def test_liveobjects_names_are_all_importable(self): + for name in server_liveobjects.__all__: + assert getattr(server_liveobjects, name) is not None diff --git a/test/unit/websockettransport_test.py b/test/unit/websockettransport_test.py index d45c3ca8..3cf1fffb 100644 --- a/test/unit/websockettransport_test.py +++ b/test/unit/websockettransport_test.py @@ -1,7 +1,7 @@ from unittest.mock import MagicMock, patch -from ably.transport.websockettransport import WebSocketTransport -from ably.types.options import Options +from ably.pubsub.transport.websockettransport import WebSocketTransport +from ably.pubsub.types.options import Options def _connect_url(host='example.com', **option_kwargs): @@ -9,7 +9,7 @@ def _connect_url(host='example.com', **option_kwargs): connection_manager.options = Options(auth_token='foo', **option_kwargs) transport = WebSocketTransport(connection_manager, host, {'format': 'json'}) with patch.object(transport, 'ws_connect', MagicMock()) as mock_ws_connect: - with patch('ably.transport.websockettransport.asyncio.create_task') as mock_create_task: + with patch('ably.pubsub.transport.websockettransport.asyncio.create_task') as mock_create_task: mock_create_task.return_value = MagicMock() transport.connect() return mock_ws_connect.call_args[0][0] diff --git a/test/uts/README.md b/test/uts/README.md index acab71e5..2a7e5976 100644 --- a/test/uts/README.md +++ b/test/uts/README.md @@ -20,12 +20,13 @@ helpers/ shared infrastructure the specifications assume, and its own tests assets/ fixtures the specifications name, vendored from elsewhere rest/ specifications under uts/rest realtime/ specifications under uts/realtime +objects/ specifications under uts/objects (LiveObjects), with helpers/ of its own ``` Every directory holding tests needs an `__init__.py`, because `test` is a package. Unit tests serve every request from a mock and reach no network — neither the REST -suite nor the realtime one. The seams are installed per client, so a test holds to +suite, the realtime one nor the objects one. The seams are installed per client, so a test holds to that by installing them; one that omits a seam, or that lets the host fallback loop run, reaches the real internet. Integration tests run against a sandbox app. @@ -40,7 +41,7 @@ mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), on_request=lambda req: req.respond_with(200, {'result': 'ok'}), ) -ably = AblyRest(key=key, _test_options=TestOptions(http_transport=mock_http.as_transport())) +ably = create_http_client(key=key, _test_options=TestOptions(http_transport=mock_http.as_transport())) ``` A realtime client takes its websocket mock the same way, through @@ -51,9 +52,9 @@ A realtime client takes its websocket mock the same way, through mock_ws = MockWebSocket( on_connection_attempt=lambda conn: conn.respond_with_success(CONNECTED_MESSAGE), ) -ably = AblyRealtime(key=key, auto_connect=False, - _test_options=TestOptions(websocket_connect=mock_ws.as_connect(), - clock=FakeClock())) +ably = create_realtime_client(key=key, auto_connect=False, + _test_options=TestOptions(websocket_connect=mock_ws.as_connect(), + clock=FakeClock())) ``` `rest_client(mock_http, ...)` and `realtime_client(mock_ws, ...)` in @@ -86,10 +87,11 @@ that closes its own leaves nothing to clean up if it fails first. | [helpers/presence.py](helpers/presence.py) | the presence-map stubs and wire-message builders the presence specifications share | | [helpers/sandbox.py](helpers/sandbox.py) | the sandbox app the integration tier provisions, the presence-fixture cipher, `random_id()` and the JWT signing the auth specification asks a library for | | [helpers/deviations.py](helpers/deviations.py) | the `@deviation` and `@spec_error` gates | +| [objects/helpers/standard_test_pool.py](objects/helpers/standard_test_pool.py) | the LiveObjects fixtures, from `uts/objects/helpers/standard_test_pool.md`; see the objects tier below | `SKILL.md` lists every name in each. The helpers have their own tests -(`helpers/*_test.py`), which are not derived from a specification and are not counted -in the derived-test totals. +(`helpers/*_test.py`, `objects/helpers/standard_test_pool_test.py`), which are not derived +from a specification and are not counted in the derived-test totals. ## The integration tier @@ -253,6 +255,55 @@ the session; the modules here sign an Ably JWT locally instead, through a file-l nothing to the log a test is counting. Tests in this package are given 300 seconds each, as in the REST proxy package. +## The objects tier + +`objects/` holds the LiveObjects specifications, laid out as the other two: +`objects/unit/`, fifteen specifications that reach no network; `objects/integration/`, three +against the sandbox; and `objects/integration/proxy/`, one through `uts-proxy`. Seven of the +unit specifications are pure — they build `InternalLiveCounter`, `InternalLiveMap`, +`ObjectsPool` or a channel-less `RealtimeObject()` and connect nothing — and the other eight +drive `channel.object` over the mock websocket. The specifications are written against an +untyped `PathObject` and `Instance`; the tests reach a type's methods through LODR-061's +views, `root.get('score').as_live_counter().value()`, which [deviations.md](deviations.md) +explains along with the shapes the pure tier adapts to (S-1 to S-5). + +Everything the objects specifications share is in +[objects/helpers/standard_test_pool.py](objects/helpers/standard_test_pool.py), named as +`standard_test_pool.md` names it: + +| Helper | Is | +|---|---| +| `setup_synced_channel(channel_name='test', mock_ws=None, clock=None, modes=OBJECTS_MODES, **client_options)` | the specifications' synced channel: a client on `standard_mock_websocket()`, the channel with both object modes, and `await channel.object.get()`. Unpacks as `client, channel, root, mock_ws`. `setup_synced_channel_no_ack` records OBJECT messages without ACKing them | +| `standard_mock_websocket(auto_ack=True, on_object=None, connected=None, ...)` | answers an ATTACH with an ATTACHED and an OBJECT_SYNC of `STANDARD_POOL_OBJECTS`, a DETACH with a DETACHED, and each OBJECT, after `on_object`, with an ACK | +| `objects_client(mock_ws, clock=None, mock_http=None, **kwargs)` | the client underneath: it connects on its own, speaks JSON, and has `GET /time` answered by `time_mock_http(clock)`, since creating an object reads the server time | +| `objects_channel_options(*modes)`, `objects_connected_message(...)`, `objects_attached_message(channel, channel_serial, flags)` | the channel options and harness messages. Granted modes are `flags` bits, `HAS_OBJECTS \| OBJECT_SUBSCRIBE_FLAG`, not a `modes` list | +| `build_counter_inc`, `build_map_set`, `build_map_remove`, `build_map_clear`, `build_object_delete`, `build_counter_create`, `build_map_create`, `build_object_state`, `build_object_message`, `build_object_sync_message`, `build_ack_message`, `json_value`, `bytes_value` | the builders, each returning the JSON-wire dictionary `send_to_client` takes: camelCase keys, numeric actions, `json` values as JSON strings, `bytes` as base64 | +| `ack_serial(msg_serial, index)`, `remote_serial(index)`, `below_ack_serial(index)`, `POOL_SERIAL` | serials that sort where the specifications need them. A bare `'99'` sorts before `POOL_SERIAL` and is rejected as stale | +| `object_message(wire)`, `object_messages(protocol_message)`, `capture_updates(obj)` | the pure tier's: a builder's output decoded to the internal type, and the updates an object emits, since `apply_operation` returns a boolean | +| `build_public_object_message(message, channel_name)` | the public `ObjectMessage` a subscription event should carry, built independently of the library | +| `assert_unchanged_after_quiescence(count_under_test, control_delivered)` | the specifications' negative-assertion pattern | +| `provision_objects_via_rest(api_key, channel_name, operations)` | REST provisioning for the integration tier, over `X-Ably-Version: 6` | + +The client reads frames on a task of its own, so a test reads state after a `poll_until` on +the frame's effect, never straight after `send_to_client`, and asserts a negative or an exact +count only once a positive control behind it has arrived and the loop has settled. + +```python +async def test_rtpo17_increment_delegates_to_counter(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().increment(25) + + assert root.get('score').as_live_counter().value() == 125 +``` + +`objects/integration/` provisions its own sandbox app from its own `conftest.py`, separate +from the realtime tier's but under the same fixture name, `realtime_sandbox`. All three of +its specifications carry Protocol Variants, so every test takes `use_binary_protocol`, and +each is given 120 seconds, as in the other integration tiers. Its `proxy/` package repeats +the realtime proxy package's `proxy_control` and `proxy_session` fixtures and 300-second +timeout, and its clients sign an Ably JWT locally, as the realtime proxy modules do. + ## Running ``` @@ -262,15 +313,16 @@ uv run --frozen --extra crypto --extra dev pytest test/uts -q The offline tiers alone, which need no network: ``` -uv run --frozen --extra crypto --extra dev pytest test/uts/rest/unit test/uts/realtime/unit test/uts/helpers -q +uv run --frozen --extra crypto --extra dev pytest test/uts/rest/unit test/uts/realtime/unit test/uts/objects/unit test/uts/helpers test/uts/objects/helpers -q ``` -Either integration tier alone, each of which provisions a sandbox app and needs network +Any integration tier alone, each of which provisions a sandbox app and needs network access: ``` uv run --frozen --extra crypto --extra dev pytest test/uts/rest/integration -q uv run --frozen --extra crypto --extra dev pytest test/uts/realtime/integration -q +uv run --frozen --extra crypto --extra dev pytest test/uts/objects/integration -q ``` `--frozen` is required: without it dependency resolution reaches past the diff --git a/test/uts/deviations.md b/test/uts/deviations.md index bf08ee74..28535f9f 100644 --- a/test/uts/deviations.md +++ b/test/uts/deviations.md @@ -24,35 +24,43 @@ One Test ID can become more than one derived test: five Test IDs in `rest/unit` `error_types_test.py`, `fallback_test.py`, `rest_client_test.py` (two) and `paginated_result_test.py` — assert several independent things under a single id, and the derivation writes a function for each rather than one function with an unrelated -second half. That turns 1132 Test IDs into 1141 derived tests. Going the other way, one +second half. That turns 1472 Test IDs into 1481 derived tests. Going the other way, one derived test can become more than one case: five of the twelve `rest/integration` -specifications and five of the twenty `realtime/integration` ones carry a `## Protocol -Variants` section and run every one of their tests twice, once per protocol, and nine -`rest/unit` tests are parametrized over a table of fixtures the specification gives -inline. That turns 1141 derived tests into 1234 pytest cases. - -Of **1132 Test IDs, derived as 1141 tests and run as 1234 pytest cases**: 904 Test IDs -(913 tests, 1001 cases) pass, 213 (213 tests, 218 cases) are gated behind +specifications, five of the twenty `realtime/integration` ones and the three +`objects/integration` ones outside `proxy` carry a `## Protocol Variants` section and run +every one of their tests twice, once per protocol; nine `rest/unit` tests are +parametrized over a table of fixtures the specification gives inline; and five +`objects/unit` tests are parametrized, three over a table and two over the scenarios the +specification loops through. That turns 1481 derived tests into 1608 pytest cases. + +Of **1472 Test IDs, derived as 1481 tests and run as 1608 pytest cases**: 1289 Test IDs +(1298 tests, 1423 cases) pass, 168 (168 tests, 170 cases) are gated behind `RUN_DEVIATIONS`, and 15 (15 tests, 15 cases) cannot be run at all. The three groups are -disjoint: two Test IDs, and one parametrized test, have a gated part and a passing part, -and are counted with the gated. Every gated test has been confirmed to fail when -enabled, so none of them passes under both behaviours. 494 of the Test IDs come from -`uts/rest/unit` (503 tests, 536 cases), 481 from `uts/realtime/unit` (481, 481), 84 -from `uts/rest/integration` (84, 122) and 73 from `uts/realtime/integration` (73, 95); 8 -of the REST integration ids (8, 8) and 30 of the realtime ones (30, 30) come from the -`proxy` package within each. Of the gated Test IDs 122 are REST and 91 realtime, which is -126 REST cases and 92 realtime. -A further 122 pytest cases under `helpers/` cover the mock infrastructure itself and are -not derived from a specification. - -The 203 gated Test IDs that record SDK non-compliance — 203 tests, 208 cases — reduce to -**71 distinct root causes**, 27 on the REST side and 44 on the realtime side. Three further +disjoint: two Test IDs have a gated part and a passing part — `TI1`, as two functions, and +`TO/endpoint-affects-host`, as one parametrized test — and are counted with the gated, +while their passing parts count among the passing tests and cases. Every gated test has +been confirmed to fail when enabled, so none of them passes under both behaviours. 494 of +the Test IDs come from `uts/rest/unit` (503 tests, 536 cases), 482 from +`uts/realtime/unit` (482, 482), 322 from `uts/objects/unit` (322, 344), 84 from +`uts/rest/integration` (84, 122), 73 from `uts/realtime/integration` (73, 95) and 17 from +`uts/objects/integration` (17, 29); 8 of the REST integration ids (8, 8), 30 of the +realtime ones (30, 30) and 5 of the objects ones (5, 5) come from the `proxy` package +within each. Of the gated Test IDs 79 are REST and 89 realtime, which is 80 REST cases +and 90 realtime. None of the 339 objects Test IDs is gated or unrunnable. +A further 148 pytest cases cover the harness itself and are not derived from a +specification: 130 under `helpers/`, for the mocks, and 18 under `objects/helpers/`, for +the objects builders and the synced-channel mock. + +The 157 gated Test IDs that record SDK non-compliance — 157 tests, 159 cases — reduce to +**69 distinct root causes**, 26 on the REST side and 43 on the realtime side. Three further defects are recorded below with no test of their own, because the specification's test cannot discriminate (RTP18a), has nothing to assert against (the timezone split on synthesized LEAVE timestamps), or is worked around in the setup of every test that would otherwise trip over it (`enterClient` on an anonymous connection), so the file -carries **74 SDK root causes** in all. The remaining 10 gated Test IDs are -specification faults, and reduce to 7. +carries **72 SDK root causes** in all. The remaining 11 gated Test IDs are +specification faults, and reduce to 8. The objects tier records no SDK non-compliance; +its specification faults are all corrected or worked around in tests that pass, and +reduce to the seven candidate issues O.1–O.7, none of them filed. Entries closed by a fix are removed rather than kept as history; `git log` holds that. @@ -75,7 +83,7 @@ from the specification text, so that correcting the specification is all it take it pass, and it is marked `@spec_error` — a skip gated on `RUN_DEVIATIONS`, the same gate `@deviation` uses, with a reason naming the specification rather than the SDK. The suite stays green, a real regression still shows, and the failure is one environment variable -away. Each is filed upstream, in the issues named below. Ten tests are gated +away. Each but RSC7d's is filed upstream, in the issues named below. Eleven tests are gated this way: | Test | Spec error | @@ -90,6 +98,7 @@ this way: | `test_rsp4_history_pagination` | RSP4 - wire action 4 asserted to be LEAVE | | `test_tp3_presence_to_json` | TP3 - an outgoing action asserted as the string `"enter"` | | `test_tp3_null_attributes_omitted` | TP3 - the same outgoing string assertion | +| `test_rsc7d_ably_agent_header_format` | RSC7d - a library-name pattern that rejects hyphens | Where instead only a specification's *fixture*, *setup* or *label* is at fault, the assertion it carries still stands. Those tests keep the corrected fixture (or the @@ -98,7 +107,8 @@ site. The entries below cover both kinds and say which applies. Almost every rea fault is of the second kind, which is why only one realtime test is gated as a spec error while fourteen `realtime/unit` entries appear below. The eight `realtime/integration` faults are of that kind without exception, so none of them is -gated either. +gated either, and so is every `uts/objects` fault: the objects entries close this +section, and not one objects test is gated. The three sections that follow this one record SDK behaviour rather than specification faults. @@ -130,6 +140,7 @@ Raised upstream: | [#552](https://github.com/ably/specification/issues/552) | `proxy/connection_resume.md`: a status code neither SDK returns, a proxy substitution that does not exist, and event-log fields the proxy does not emit | | [#553](https://github.com/ably/specification/issues/553) | A heartbeat-starvation test that closes the socket thirteen seconds inside the idle window | | [#554](https://github.com/ably/specification/issues/554) | Two sections provoking one server response, leaving the revoked-key point uncovered | +| [#559](https://github.com/ably/specification/issues/559) | Batch publish and token revocation fixtures in the response format the server sends below protocol version 3 | `#527` also carries a comment on the realtime wire-format assertions, `#532` one on the same housekeeping categories in `realtime/unit`, and @@ -139,11 +150,14 @@ RSA4c3 contradiction, since that issue is what decides it. `#547` to `#550` are the `uts/rest/integration` faults and `#551` to `#554` the `uts/realtime/integration` ones. Each of them is of the second kind — a fixture, a setup step or a header label — so the derived test keeps the corrected fixture and passes, and -none of them is among the ten gated above. +none of them is among the eleven gated above. Not every entry has an issue of its own: the URL-safe base64 alphabet is recorded below -and not filed, because ably-python's own encoding settles the tests either way. Line -references in these entries are against `ably/specification@d9a04ca`. +and not filed, because ably-python's own encoding settles the tests either way. None of +the `uts/objects` faults is filed yet; *Candidate issues* classifies them as O.1–O.7 for a +maintainer to file. Line references in these entries are against +`ably/specification@d9a04ca`, and in the `uts/objects` entries against +`ably/specification@747796f`. ### `/time` is stubbed as an object rather than an array @@ -153,6 +167,15 @@ references in these entries are against `ably/specification@d9a04ca`. returns a one-element array, which is what `time.md` itself uses and what RSC16 describes. Any SDK that indexes the array raises. +### RSC7d's Agent pattern rejects a hyphenated library name + +`rest_client.md`'s `RSC7d/ably-agent-header-format-0` asserts that the `Ably-Agent` +header matches `ably-[a-z]+/[0-9]+\.[0-9]+\.[0-9]+`. RSC7d1 asks only for a series of +`key[/value]` entries that include the library's name and version, and puts no limit on +the characters a key may hold. ably-python reports itself as `ably-pubsub-python/`, +which RSC7d1 allows and the pattern rejects. The test is gated with `@spec_error`. The +fault is not filed upstream yet. + ### Spec points are mislabelled across four specs The assertions are sound; the points they are filed under are not. @@ -261,16 +284,36 @@ file, and the protocol, which fix LEAVE at 3 and UPDATE at 4. The closing note o `batch_presence.md` states that with `X-Ably-Version >= 3` the server returns a `BatchResult` envelope "for all batch responses" and calls the plain array legacy. -Every mock in `batch_publish.md` uses the plain array. `features.md` RSC22b backs -`batch_publish.md` — "the response will still be an array" — so `batch_presence.md`'s -claim is the one to revisit. `revoke_tokens.md` has the same internal split: -`RSA17c_1` and `TRS2_1` stub a bare array while asserting envelope fields. +Every mock in `batch_publish.md` uses the plain array, and its RSC22_Headers1 pins +`X-Ably-Version: 2`: `batch_publish.md` is written against the legacy protocol, and +`batch_presence.md` is the one that matches the server ably-python talks to. Measured +against the sandbox, `POST /messages` answers: -`batch_publish.md` RSC22_Headers1 also pins `X-Ably-Version: 2` and -`Content-Type: application/json`; CSV2b templates the version, the sibling spec says -">= 3", and the binary protocol default makes the content type msgpack. - -The revocation half of that split is settled by the server. `POST +| `X-Ably-Version` | Every channel succeeds | Any channel fails | +|---|---|---| +| 5 | 201 with an array holding one `{successCount, failureCount, results}` envelope per spec | the same, 201 | +| 2, or none | 201 with one flat array of `{channel, messageId}` across every spec | 400 with error 40020, and the flat array, failures included, as `batchResponse` | + +A spec sent as a bare object is answered like an array of one: an array holding a single +envelope. That is the array RSC22b means when it says the response "will still be an array" +and the single-spec overload "will have to extract the element" — an array of +`BatchResult`s, one per spec, rather than of per-channel results. `GET /presence` follows the +same split by version, answering a mixed batch with 200 and the envelope at version 5 and +with 400/40020 and `batchResponse` at version 2. The legacy shape carries no `serials` +either; `batch_publish.md`'s mocks add them to it. + +ably-python sends version 5, so every `batch_publish.md` mock is corrected to the envelope, +through `batch_result()` in `rest/unit/batch_publish_test.py`, and the assertions stand as +written: they read `result.results[...]`, the layout BAR2c gives the envelope. RSC22_Headers1's +two pins are corrected beside them — CSV2b templates the version, which ably-python sends as +5, and the binary protocol default (TO3f) makes the content type msgpack, as +[#527](https://github.com/ably/specification/issues/527) records for the unit specs that +read a body without pinning the protocol. The envelope disagreement is filed as +[#559](https://github.com/ably/specification/issues/559), together with the two +`revoke_tokens.md` mocks below. + +`revoke_tokens.md` has the same internal split: `RSA17c_1` and `TRS2_1` stub a bare array +while asserting envelope fields. The revocation half is settled by the server too. `POST /keys/{keyName}/revokeTokens` with `X-Ably-Version: 5` answers 201 with the `{successCount, failureCount, results}` envelope `revoke_tokens.md`'s "Server Response Format" section describes, for a mixed success/failure batch as well as an all-success @@ -390,7 +433,7 @@ the site. | RTL4b (`channel_attach.md`) | `channelRetryTimeout: 100` ("short timeout for testing"), then `AWAIT_STATE client.connection.state == suspended` | `channelRetryTimeout` governs channel retries, not the connection's suspend timer, which runs for `connectionStateTtl`. The test does not enable fake timers either, so on real time it would wait out two minutes. Derived with a `FakeClock`, passing the named option through unchanged | | RTP5f, RTL11 (`realtime_presence_channel_state.md`) | `simulate_disconnect()` then `AWAIT_STATE channel.state == suspended` | A transport drop reaches DISCONNECTED. RTL3c propagates SUSPENDED to channels only from a SUSPENDED *connection*, so the awaited state never arrives. RTP5f's own note ("e.g. connection transitions to SUSPENDED") says as much; the steps do not carry it out | | RTP5a (`realtime_presence_channel_state.md`) | detach, then `presence.get(waitForSync: false).length == 0` | RTP11e has `get` run the ensure-active-channel procedure for any state but SUSPENDED, so the read-back re-attaches the channel and the specification's own server then repopulates the very map being checked for emptiness. The derived test reads the two maps directly | -| RTS3c1, RTL16a (`channel_options.md`), RTS4a (`channels_collection.md`) | `autoConnect: false`, no mock installed, `connect()` never called, then `AWAIT channel.attach()` and assert ATTACHED | RTL4b requires `attach()` to fail unless the connection is CONNECTING, CONNECTED or DISCONNECTED, and with no mock nothing would answer the ATTACH in any case. Derived with a mock that connects and answers each ATTACH. Upstream should give these three setups a mock, as the sibling sections of the same files do | +| RTS3c1, RTL16a (`channel_options.md`) | `autoConnect: false`, no mock installed, `connect()` never called, then `AWAIT channel.attach()` and assert ATTACHED | RTL4b requires `attach()` to fail unless the connection is CONNECTING, CONNECTED or DISCONNECTED, and with no mock nothing would answer the ATTACH in any case. Derived with a mock that connects and answers each ATTACH. Upstream should give these setups a mock, as the sibling sections of the same file do | | RTS3c1 `error-reattach-modes-1` (`channel_options.md`) | `# Put channel in attaching state (implementation detail)` | The premise the test turns on is the one step it does not give, and the setup has no mock to reach ATTACHING with | ### Fixtures written against mock methods the contract does not define @@ -763,18 +806,218 @@ Filed as [#554](https://github.com/ably/specification/issues/554). | `message_encoding.md`, `msgpack_interop.md`, `annotations.md` | Six sections carry no Test ID; ids were inferred by sibling convention | | `publish.md`, `rest_presence.md`, `message_encoding.md`, `history.md`, `idempotency.md` | All point at `/Users/paddy/data/worknew/dev/dart-experiments/...` for the mock contract | | `revoke_tokens.md` | The all-success Setup blocks stub HTTP 200 with a plain array, the shape the file's own "Server Response Format" section calls legacy and says no current SDK sees. The `BatchResult` envelope and HTTP 201 that section prescribes appear only in the mixed and all-failure blocks | -| `publish.md` (integration) | The `Spec points:` header reads RSL1d, RSL1l1, RSL1m4, RSL1n, and the file carries a fifth section, `## RSL1k5 - Idempotent publish with client-supplied IDs`, with its own Test ID. The section is sound; only the header is short. Same housekeeping class as [#532](https://github.com/ably/specification/issues/532) | -| `auth.md` (integration) | RSC10's expired-JWT fixture is `generate_jwt(expires_at: now() - 5_seconds)`, naming `exp` and leaving `iat` open. Ably reads a JWT's lifetime as `exp - iat` and rejects a negative one with 400/40003 "Invalid value for ttl" before it considers expiry, so `iat` at now produces a token that fails the wrong way and never reaches the 40140–40149 renewal path the test is about. Backdating `iat` past `exp` gives the already-expired token the test wants, answered 401/40142. An SDK signing its own Ably JWT has to choose, so the fixture should say which | -| `batch_presence.md` | BGR2 says a channel with no members "returns a success result with an empty `presence` array", and the unit tier's mocks all send `'presence': []`. The server sends no `presence` key at all, so an implementation has to default the field for the assertion to hold. The derived test asserts the specification's `length == 0`, with the wire shape in a comment | | `publish.md` (integration) | The `Spec points:` header reads RSL1d, RSL1l1, RSL1m4, RSL1n, and the file carries a fifth section, `## RSL1k5 - Idempotent publish with client-supplied IDs`, with its own Test ID. The section is sound; only the header is short. `auth.md` (integration) has the same shape: its header reads RSA4, RSA8 and it carries `## RSC10` with its own Test ID. Same housekeeping class as [#532](https://github.com/ably/specification/issues/532); filed as [#550](https://github.com/ably/specification/issues/550) | | `auth.md` (integration) | RSC10's expired-JWT fixture is `generate_jwt(expires_at: now() - 5_seconds)`, naming `exp` and leaving `iat` open. Ably reads a JWT's lifetime as `exp - iat` and rejects a negative one with 400/40003 "Invalid value for ttl" before it considers expiry, so `iat` at now produces a token that fails the wrong way and never reaches the 40140–40149 renewal path the test is about. Backdating `iat` past `exp` gives the already-expired token the test wants, answered 401/40142. An SDK signing its own Ably JWT has to choose, so the fixture should say which. Filed as [#550](https://github.com/ably/specification/issues/550) | | `batch_presence.md` | The restricted-key setup's comment reads "only has access to \"batch-allowed\" channel" while the setup fixes `allowed_channel = "channel6"`; `batch-allowed` appears nowhere in the file. Filed with [#548](https://github.com/ably/specification/issues/548), whose fix replaces the same lines | | `batch_presence.md` | BGR2 says a channel with no members "returns a success result with an empty `presence` array", and the unit tier's mocks all send `'presence': []`. The server sends no `presence` key at all, so an implementation has to default the field for the assertion to hold. The derived test asserts the specification's `length == 0`, with the wire shape in a comment. Filed as [#550](https://github.com/ably/specification/issues/550) | +### The objects unit specifications model three internal interfaces against `objects-features.md` + +**Spec points:** RTLC7, RTLM15, RTO3–RTO9, RTO17, RTLCV4g5, RTLMV4j5, PAOOP3. Not yet +filed; candidate O.1. + +A conforming SDK passes these only by adapting, so each is also a shape adaptation under +*Adapted Tests*, which defines the labels S-1, S-2 and S-4. + +- `internal_live_counter.md` and `internal_live_map.md` read + `update = obj.applyOperation(...)` and, of the same call elsewhere in the same files, + assert `result == true` or `false`. RTLC7g and RTLM15g make the return a boolean and + have the update *emitted* (RTLC7d1a, RTLM15d1a and their siblings), so the files + contradict themselves and `objects-features.md` settles it. (S-1) +- `objects_pool.md`, and the three RTO5c10 cases of `parent_references.md`, put the sync + and apply state machine — `processAttached`, `processObjectSync`, + `processObjectMessage`, `applyObjectMessages`, `syncState` — on `ObjectsPool`, and build + `RealtimeObject(pool: pool)`. RTO3a makes the pool a `Dict`, and + RTO7a, RTO7b and RTO17 give the sync state, the buffer and `appliedOnAckSerials` to + `RealtimeObject`. `uts/objects/PLAN.md` repeats the pool model. (S-2) +- `value_types.md` reads the create an evaluated value type retains as + `operation.counterCreate` / `operation.mapCreate`, and `public_object_message.md` reads + it as `derivedFrom` on the `*CreateWithObjectId` payload. RTLCV4g5 and RTLMV4j5 say only + that it is retained alongside, locally, and is not sent — which the first reading would + put on the wire. The same file's PAOOP3b2 and PAOOP3c2 sources give + `mapCreateWithObjectId` the fields `objectId`, `semantics` and `entries`, and + `counterCreateWithObjectId` `objectId` and `count`, where MCRO2 and CCRO2 define only + `initialValue` and `nonce`. (S-4) + +**Tests affected:** the 42 tests in `internal_live_counter_test.py` (13) and +`internal_live_map_test.py` (29) that read an update or a `false`; 30 in +`objects_pool_test.py` and 3 in `parent_references_test.py`; 8 in `value_types_test.py`; +and `test_paoop3_map_create_from_with_object_id` and +`test_paoop3_counter_create_from_with_object_id`, which build the payload MCRO2 and CCRO2 +describe, with `initialValue` the JSON of the derived payload and the derived payload in +`derived_from`. All pass. + +### Objects fixtures that put a value where the protocol does not + +**Spec points:** RTO2, RTO25a, RTO26a, RTLMV4d, RTLMV4f1, and `standard_test_pool.md`. Not yet +filed; candidate O.2. + +| Spec | Fixture | Why it cannot hold | Derived as | +|---|---|---|---| +| `realtime_object.md` RTO2 `mode-enforcement-0`, RTO25a `access-requires-subscribe-mode-0`, RTO26a `write-requires-publish-mode-0` | An ATTACHED carrying `modes: ["OBJECT_SUBSCRIBE"]` | The modes a server grants travel as bits of the ATTACHED's `flags` (RTL4m, TR3), which is where a client decodes them; a `modes` field is read by nothing | `flags: HAS_OBJECTS \| OBJECT_SUBSCRIBE_FLAG`, through `objects_attached_message(..., flags=...)` | +| `standard_test_pool.md`'s CONNECTED, and the hand-written ones in `internal_live_counter_api.md` and `internal_live_map_api.md` | `connectionId` inside `connectionDetails` | `connectionId` is a ProtocolMessage field, which is where a client reads it | `objects_connected_message()` puts it on the message | +| `value_types.md` `RTLMV4d/map-set-all-types-table-0` | Each row compares the in-memory MapCreate's entry with a decoded value — the json rows expect `[1, "a", null]` and `{"k": "v"}` — except the bytes row, whose `"AQID"` is the JSON wire's base64 | In memory the entry holds bytes; base64 is what `initialValue` carries (RTLMV4f1). ably-js's derived test reads `initialValue` for that reason | `bytes([1, 2, 3])` on the retained MapCreate and `"AQID"` in the decoded `initialValue`, so both readings are pinned. The row should read `bytes([1, 2, 3])` | +| `standard_test_pool.md`'s `provision_objects_via_rest` | "a JSON array of result objects (one per batch entry)" | Measured: one result object for the whole batch, whose `objectIds` lists every object created or updated. `X-Ably-Version` 5 and 6 both accept the v6 operation format | `provision_objects_via_rest` flattens either shape | + +### Objects mock-tier steps read state as though an injected frame were applied at once + +**Spec points:** RTO5c9, RTO10c1b1, RTO17, RTO18d, RTO19, RTO20, RTO20d4, RTO20e, RTO20e1, +RTO23c1, RTO24b1, RTO24c1, RTINS16, RTINS16h, RTLO4b, RTPO19, and at the integration tier +RTO5a2, RTO7, RTO8 and RTO17. Not yet filed; candidate O.3. + +`path_object.md`'s own compact tests say that an SDK may apply an inbound OBJECT message +asynchronously, and wait for it; `standard_test_pool.md` defines a quiescence pattern for +negative assertions. The steps below use neither, and are written as though +`send_to_client` returned with the frame applied. A client that reads frames on a task of +its own, as ably-python does, meets four shapes: + +| Shape | Where | What it costs | +|---|---|---| +| A read or a negative straight after `send_to_client` | `RTO20/echo-dedup-0`, `RTO5c9-RTO20/ack-serials-cleared-on-resync-0`, `RTO19/off-deregisters-0`, `RTO10c1b1/gc-root-never-removed-0`; and `RTINS16h/subscribe-no-side-effects-0`, straight after `subscribe` | Each reads the state from before the frame, so the negatives — no double apply, no callback, the root not removed, no side effect — pass whether or not the SDK is right | +| An operation started before the ATTACHED that should hold it back has been processed | the three `RTO23c1` tests, the two `RTO20e1` tests, `RTO20e/waits-for-synced-0`, `RTO20d4/empty-synthetic-list-skips-sync-wait-0` | `get()` or the write still sees SYNCED and resolves at once, so a **conforming** SDK fails `IS NOT complete`. ably-js's derived tests insert `flushAsync()` at that point | +| A seed sent on the line before the subscription it must not reach | `RTO24c1/depth-filtering-formula-0`, `RTO24b1/multi-path-dispatch-0` | A seed applied after the subscription is delivered to it, and `events.length == 2` reads 3, `events_alias.length == 1` reads 2 | +| An exact count straight after `poll_until(count >= n)` | throughout `live_object_subscribe.md` and `path_object_subscribe.md`, `RTINS16/subscribe-receives-events-0`, and `RTO18d`, `RTO24c1` and `RTO17-RTO18` in `realtime_object.md` | The "exactly" half is a negative, read before a second, wrong event could arrive | + +`objects_faults.md` has the integration tier's version. `RTO5a2-RTO17/sync-interrupted-reconnect-0` +and `RTO7-RTO8/mutations-buffered-during-resync-0` `AWAIT_STATE` DISCONNECTED, which RTN15a +leaves at once — `writing-test-specs.md`'s *Verifying Transient States* says not to — and then +CONNECTED, which the client may still hold from before the drop. `RTO17/server-detach-resync-0` +awaits ATTACHED on a channel that has not yet processed the DETACHED injected onto it, so the +wait returns at once and says nothing about the re-attach. + +The derived tests keep every assertion and add the barrier: a `poll_until` on the frame's +effect before a read; for a negative, a positive control delivered behind the message under +test and `assert_unchanged_after_quiescence` (echo-dedup's control is an increment of the +nested counter, RTO10c1b1's an OBJECT_DELETE of the score counter, RTO19's a second SYNCED +listener); `_restart_sync`, which polls until `_sync_state` is SYNCING before the operation +starts; a poll on the seeded key before subscribing; and `await settle()` between a positive +poll and an exact count. The integration tests wait on a state recorder registered before the +fault, as the realtime tier does. The specifications want `process_pending_events()` after +each injected frame they then read from, and their own quiescence pattern before each negative. + +### Objects tests that pass without the behaviour they are named for + +Not yet filed; candidate O.4, the same class as +[#543](https://github.com/ably/specification/issues/543). + +Where the fix is an assertion the specification omits, the derived test carries every one of +the specification's assertions and adds it, with a `# UTS SPEC ERROR:` comment or a comment +citing the spec point at the site — the RSL2b3 precedent above. "A mutant passed" means +measured: a throwaway reference implementation with that single fault passed every assertion +the specification makes. + +| Test ID | Why it cannot fail | What the derived test adds | +|---|---|---| +| `RTLM8g/map-remove-clear-timeserial-floor-0` | The entry's own timeserial, `"04"`, is later than the MAP_REMOVE's `"03"`, so RTLM8a1 and RTLM9e reject it whether or not the RTLM8g floor exists. A mutant ignoring the floor passed | A MAP_REMOVE of a key with no entry at `"04"`: at or below the floor and newer than the site's serial, so only RTLM8g can reject it | +| `RTLM9b/both-empty-reject-0` | Its own note: RTLO4a3 rejects the empty `ObjectMessage.serial` first, so RTLM9b is never reached | `InternalLiveMap.can_apply_map_operation('', '')` and `(None, None)` are False, checked directly | +| `RTO10/gc-tombstoned-objects-0`, `RTO10c1b1/gc-root-never-removed-0` | `score` reads null from the moment the OBJECT_DELETE applies (RTLM14c, RTLM5d2h), so both pass with no GC sweep at all | That the counter has left `_objects_pool`; for RTO10c1b1, a control OBJECT_DELETE whose removal shows a sweep ran | +| `RTO10b1/gc-grace-period-source-0` | As RTO10, and it advances 6000 ms, less than one GC interval — RTO10a's example is five minutes — so no sweep runs. Its stated claim, that it fails if `objectsGCGracePeriod` is ignored, cannot hold | `_gc_interval_ms = 1000` before `get()`, `_gc_grace_period_ms == 5000`, and the counter still in the pool at 4000 ms and gone at 6000 ms | +| `RTPO6/at-escaped-dots-0` | `at("a\\.b.c").path() == "a\\.b.c"` holds for an `at` that splits on every dot too, since `path()` renders `['a\\', 'b', 'c']` back the same way | `_path == ['a.b', 'c']` | +| `RTPO19/map-clear-triggers-child-events-0` | The MAP_CLEAR's `"99"` sorts before the pool's `"t:0"`, so RTLM24e1 removes nothing; and nothing subscribes at a child path, which is what the title is about | The clear at `remote_serial(0)`, and a subscription at `name` that must receive an event for `name` | +| `RTO4b/attached-no-objects-synced-0` | Lists RTO4b2's "must not create a new root"; a mutant replacing the root passed | `pool['root'] is root` | +| `RTO14/objectid-format-counter-0` | Lists RTO14b1's SHA-256 of `[initialValue]:[nonce]` and asserts only the alphabet; a mutant hashing the wrong input passed | The digest, computed in the test, and that the hash decodes | +| `RTINS16g/subscription-follows-identity-0` | Nothing checks that the MAP_SET repointing `score` applied, so a rejected repoint passes | That `score` resolves to the new counter, that the event is the increment of 10, and that the counter reads 110 | +| `RTO20e/waits-for-synced-0` | `inc_future IS NOT complete` is asserted before the publish could have been sent | Pending only once the OBJECT has left the client and its ACK has been processed | +| `RTO23c/get-waits-for-synced-0` | Asserts only the eventual result, which a `get()` that does not wait also produces | `get()` still pending after the ATTACHED has been processed | +| `RTO20c/missing-site-code-0`, `RTO20d1/null-serial-skipped-0` | "The value stays 100" also holds if nothing was published | Exactly one OBJECT was published | +| `RTO20e1/fails-on-channel-detached-0`, `RTO20e1/fails-on-channel-failed-0` | RTO20e1's text gives the status code, and for FAILED the cause, and the assertions read neither | `status_code == 400`, and `cause.code == 90000` for FAILED | + +Six more are derived as written, there being no assertion to add that is not the +specification's to choose: + +- `objects_pool.md`'s `RTO4-RTO5/attached-during-syncing-resets-0` gives the second sync a new + sequence id, so RTO5a2 discards the old objects whether or not the ATTACHED reset anything; + only removing RTO5a2a's clear made it fail. `objects-features.md` does not say that an + ATTACHED with HAS_OBJECTS resets a sync in flight either; see the features entry below. +- `objects_faults.md`'s `RTO7-RTO8/mutations-buffered-during-resync-0` and + `RTO5-RTO7/publish-during-sync-echo-after-0` are titled for the buffering of RTO7 and RTO8 + and never exercise it. Measured: in the first, client B resumes about 60 ms after the + proxy's disconnect and the server delivers A's write inside the resync data, with no OBJECT + after the resume; in the second, uts-proxy's `delay` holds every later frame behind the + delayed OBJECT_SYNC, so A's OBJECT reaches B once B is SYNCED and is applied directly. + Neither can fail on a missing RTO8a buffer. +- `internal_live_counter_api.md`'s `RTLC11/counter-update-on-inc-0` cites RTLC11b1, the + update's `amount`, and asserts the inbound operation's `counterInc.number`, which is what + `Instance#subscribe`'s event carries; RTLC11b1 is covered at the pure tier only. +- `live_object_subscribe.md`'s `RTLO4b/subscribe-map-update-0` states that the update names + the key and asserts only `updates.length == 1`. +- `objects_pool.md`'s `RTO5c10/empty-sync-parent-refs-0` asserts + `pool["root"].parentReferences == {}`, which holds in every state the test can build; the + rest of the test has substance. +- `objects_lifecycle_test.md`'s `RTO23-RTPO15/set-primitive-propagates-0` subscribes + `events_b` and never reads it. + +### Boundaries no objects test reaches + +**Spec points:** RTLM7h, RTLM8g, RTLM24c, RTLM19a1, RTLC7b, RTLC7c, RTLC7e, RTLM15b, RTLM15c, +RTLM15e, RTLC6a, RTLC6e, RTLM6a, RTLM6e, RTLM7a1, RTLM7a3, RTLM7g, RTLM8a3, RTLM23c, RTLO6b, +RTINS3b. Not yet filed; candidate O.5. + +Missing tests rather than faulty ones. Each but the last was found as a single-point mutant +of a reference implementation that every test in `internal_live_counter.md` and +`internal_live_map.md` let through; the last is a point a specification's table lists and no +assertion reads. The implementation was checked against all of them outside the suite. The +derived tests do not add them, since a test with no Test ID cannot be traced back. + +| Spec point | Not covered | +|---|---| +| RTLM7h, RTLM8g | A clear timeserial **equal** to the operation's serial rejects the operation | +| RTLM24c | An equal clear serial is **not** stale; only a strictly greater one is | +| RTLM19a1 | An entry exactly `gracePeriod` old is released; the fixture uses `gracePeriod + 1` | +| RTLC7c before RTLC7e, RTLM15c before RTLM15e | A tombstoned object still records the CHANNEL serial before it rejects the operation | +| RTLC6a before RTLC6e, RTLM6a before RTLM6e | `replaceData` on a tombstoned object still replaces `siteTimeserials` | +| RTLM7a1 and RTLM7h before RTLM7a3 and RTLM7g, and RTLM8a3 | A MAP_SET or MAP_REMOVE rejected by LWW or by the clear floor keeps the old child's parent reference and creates no zero-value object | +| RTLC7b, RTLM15b | A rejected operation leaves `siteTimeserials` untouched | +| RTLM23c | A no-op entry inside a MAP_CREATE is left out of the merged update | +| RTLO6b | A MAP_REMOVE or `replaceData` tombstoning a map entry with no `serialTimestamp` reads the local clock | +| RTINS3b | `instance.md`'s `RTINS3/id-returns-objectid-0` lists it — `id` is null for a primitive — and asserts nothing about it | + +### Objects labels, commentary and harmless fixture slips + +Not yet filed; candidate O.6, the housekeeping class of +[#532](https://github.com/ably/specification/issues/532). No assertion depends on any of them. + +| Spec | Fault | +|---|---| +| `internal_live_map.md` `RTLM24/map-clear-preserves-newer-0` | The "Spec requirement" line removes entries with a timeserial `<=` the clear's serial; RTLM24e1 removes only strictly older ones, and the same file's `map-clear-basic-0` keeps the equal entry. The fixture has no equal entry | +| `internal_live_map.md` `RTLM14c/tombstoned-ref-yields-null-0` | Titled "MAP_SET referencing tombstoned objectId"; the body seeds `map.data` directly and performs no MAP_SET | +| `realtime_object.md` `RTO20d4/mixed-null-serials-applies-non-null-0` | The commentary has `child` dangle and read undefined per RTLM5d2f1. Applying the MAP_SET creates the zero-value counter (RTLM7g1) and references it (RTLM7g2), so `child` resolves to a counter reading 0. `LiveCounter.create(5)` in the fixture would let the test show the create was skipped | +| `realtime_object.md` `RTO19/off-deregisters-0` | Calls `sub.off()` on what `on()` returned, which is RTO18f2. `RealtimeObject#off` itself, RTO19, is exercised by no test in the file | +| `realtime_object.md` `RTO20/ack-after-echo-no-double-apply-0` | The requirement line credits RTO9a3 with the single apply; with the echo first it is the RTLO4a newness check on the LOCAL apply that rejects the second. The assertion is the same either way | +| `path_object.md` `RTPO6b/at-non-string-throws-0` | Asserts 40003 for a path that is not a string, where `objects-features.md`'s RTPO6 defines no error; RTPO5b gives one for `get` only. Derived as written, with a note at the site, 40003 being what RTPO5b gives the same mistake | +| `internal_live_map.md` `RTLM15e/tombstoned-reject-ops-0` | Sets `isTombstone` without `tombstonedAt`, breaking RTLO3e's invariant that one is set exactly when the other is. Kept as written; the counter twin sets both | +| `internal_live_map.md`, the pool-backed tests (RTLM7g, RTLM14, RTLM14c, every `parentReferences` test) | Build `InternalLiveMap(objectId: "root", pool: pool)` beside the root `ObjectsPool()` already holds (RTO3b1), so the map under test is a second `root`, and the parent references it writes resolve to the pool's. The module docstring says so | +| `objects_pool.md`, `parent_references.md` | Every ATTACHED carries a `channelSerial`, and `RTO5-RTO7/new-sync-keeps-buffer-0` calls `seq2:` new relative to it. RTO4 reads only the HAS_OBJECTS flag; the sync cursor is the OBJECT_SYNC's (RTO5a1) | +| `path_object_subscribe.md` | Mixes `remote_serial(n)` with bare `"98"`, `"99"`, `"100"`, `"50"` and `"serial-1"` from one site. Each applies only because no test sends two of them to one object: `"100"` sorts before `"99"` | +| `value_types.md` `RTLMV4a/evaluate-validates-entries-0` | Carries no language note for `LiveMap.create(null)`, where RTLC12e1's table and RTLMV4b both carry one; see *Adapted Tests* | +| `internal_live_map_api.md` `RTLM20/set-invalid-values-table-0` | Its invalid values — a function, `undefined`, a symbol — are JavaScript's; see *Adapted Tests* | +| `uts/docs/integration-testing.md` | Names `objects_lifecycle_test.md` and `objects_sync_test.md` as the objects specifications with Protocol Variants; `objects_gc_test.md` carries the section too, and runs both | + +### `objects-features.md` gaps the derivation and implementation met + +Not faults in a UTS specification, and not yet filed; candidate O.7. The right-hand column is +what ably-python does, and names ably-js where it agrees. + +| Spec point | Gap | ably-python | +|---|---|---| +| RTLM5d2 | Lists boolean, bytes, number, string and objectId, so an `ObjectData.json` entry falls to RTLM5d2g and reads null, though OD2g defines `json` and `path_object.md`'s compact test expects the decoded value | The decoded JSON, as ably-js | +| OD4c5, OD4d5 | Still put a JSON payload in `string` with `encoding: "json"`, where OD2g and ably-js use the `json` field, a JSON string on both wires | Sends `json`, decodes both | +| RTO4 | With HAS_OBJECTS set, does RTO4c and RTO4d only; whether a sync in flight, its SyncObjectsPool and its sequence id are abandoned is unsaid | Abandons them, as ably-js | +| RTO5a5, RTO5a6 | Whether an OBJECT_SYNC with no parseable `channelSerial` discards a sequence already accumulating | Discards it | +| RTO5d | "Skip processing that `ProtocolMessage`" when an ObjectMessage has no `object`. Skipping the ProtocolMessage loses its cursor and the sync never completes; the UTS and ably-js skip the ObjectMessage | Skips the ObjectMessage | +| RTO4b2 | Resets the root's data to RTLM4c's zero value, and is silent on `clearTimeserial` | Resets it with the data | +| RTO14b2 | Cites RFC 4648 §5, which allows padding; the UTS and ably-js require none | No padding | +| RTO2a | Checks the granted modes when they are "populated"; an ATTACHED with no mode bits decodes to none at all | No granted modes counts as unpopulated, so the requested ones are checked (RTO2b), as ably-js. A real ATTACHED always carries the bits; the standard mock's does not, nor does `objects_faults.md`'s injected `flags: 128` | +| RTPO6 | No error for a path that is not a string | 40003, as RTPO5b gives `get` | +| RTLMV4c1 | A live object, PathObject or Instance as a value is 40013 only in the unmerged [#491](https://github.com/ably/specification/pull/491) | 40013 | +| RTPO20c ([#471](https://github.com/ably/specification/pull/471)) | 92007 when the path "does not resolve to a `LiveObject`"; ably-js resolves through `_resolvePath` and gives **92005** for a path that does not resolve | 92007, as written | +| RTBC16d ([#471](https://github.com/ably/specification/pull/471)) | Publishes a batch through RTO15, so its writes would apply only on their echo | Through RTO20, applying on ACK like every other write, as ably-js | + ## Failing Tests The specification's assertion is preserved and gated behind `@deviation`. Removing -the mark is the only change needed once the SDK behaviour lands. +the mark is the only change needed once the SDK behaviour lands. Nothing derived from +`uts/objects` is gated, so no LiveObjects entry appears in this section. ### Unimplemented features @@ -782,8 +1025,8 @@ Nothing to fix here, only something to build. Each row is one feature, and the c the number of gated Test IDs that fall with it, with the pytest case count beside it where the two differ. -Five of these rows are gated at both tiers. `batchPresence`, `Auth#revokeTokens`, the -`PushChannel` surface and the `clientId` filter on `RestPresence#get` each carry +Four of these rows are gated at both tiers. `Auth#revokeTokens`, the `PushChannel` surface +and the `clientId` filter on `RestPresence#get` each carry `uts/rest/integration` tests as well as unit ones, and connection recovery carries two `uts/realtime/integration` ones; all are written against the spelling the unit tier already gates on, so both tiers go green together when the API lands. Those @@ -791,17 +1034,15 @@ integration tests do all their real work first — the sandbox app, the channels presence members entered over a realtime connection, the registered device and the issued token are all real, and each test reaches the missing call before it fails, so the assertions either side of it are known to hold against real server responses. The -`batch_presence` and `push_channels` files were additionally run against throwaway shims -— a `batch_presence` forwarding to `GET /presence`, and a `PushChannel` posting and -deleting `/push/channelSubscriptions` with `X-Ably-DeviceToken` — and pass in full -against them. The two recovery tests are a different shape: there is no missing call for +`push_channels` file was additionally run against a throwaway shim — a `PushChannel` +posting and deleting `/push/channelSubscriptions` with `X-Ably-DeviceToken` — and passes in +full against it. The two recovery tests are a different shape: there is no missing call for them to reach, so each runs end to end against the sandbox through `uts-proxy` and fails on the `recover` parameter the connection never sends. | Spec points | Missing | Test IDs | |---|---|---| -| RSC22, RSC24, BSP2, BPR2, BPF2, BAR2, BGR2, BGF2 | `batchPublish` and `batchPresence`, and all six result types. `grep -rn batch ably/` finds nothing | 44 (47 cases) | -| RSA17, RSA17b–g, BAR2, TRS2, TRF2 | `Auth#revokeTokens`, `TokenRevocationTargetSpecifier`, `BatchResult`. Gated against `auth.revoke_tokens(targets, issued_before=, allow_reauth_margin=)` returning `success_count` / `failure_count` / `results`, with `target` / `issued_before` / `applies_at` / `error` per result. RSA17d is the one case that needs no server at all — a token-authenticated client must refuse locally with 40162/401 — so it can be satisfied before any of the wire work | 21 | +| RSA17, RSA17b–g, BAR2, TRS2, TRF2 | `Auth#revokeTokens`, `TokenRevocationTargetSpecifier` and the two token revocation result types; the `BatchResult` they would arrive in is the batch API's. Gated against `auth.revoke_tokens(targets, issued_before=, allow_reauth_margin=)` returning `success_count` / `failure_count` / `results`, with `target` / `issued_before` / `applies_at` / `error` per result. RSA17d is the one case that needs no server at all — a token-authenticated client must refuse locally with 40162/401 — so it can be satisfied before any of the wire work | 21 | | RSH7, RSH7a–e, RSH6, RSH8 | `PushChannel`: `channel.push`, `client.device`, `LocalDevice`. The push *admin* surface (RSH1) does exist | 12 | | RTN16, RTN16f–k, RTC1c (TO3i) | Connection recovery, entire. `recover` is in the `Options` signature, stored, and given a property and a setter (`options.py:30,111,193,196`), and read nowhere. No `Connection#createRecoveryKey`, no `recover` connect parameter, no recovery-key decoding. Measured through the proxy: a client built with a valid `recover=` opened a `ws_connect` whose query parameters were `{'accessToken': …, 'v': '5'}` — no `recover` — and was given a fresh `connectionId`. RTN16l is otherwise fully compliant, taking the proxy's `recovery-failed-new-id`, `recovery-failed-new-key` and error 80008 and staying CONNECTED; only the absent parameter fails it | 8 | | RTL22, RTL22a–d, MFI1, MFI2a–e | `MessageFilter`. `RealtimeChannel.subscribe` (`channel.py:262-273`) accepts only a `str` or a callable, and there is no filter type of any shape to spell. Each test builds its filter through the module's `message_filter()` helper, which is the one place to repoint when the type lands | 5 | @@ -813,7 +1054,7 @@ on the `recover` parameter the connection never sends. | TP3a, TP3d, TP3g | Presence attributes defaulted from the encapsulating ProtocolMessage. There is no ProtocolMessage type; `ably/realtime/channel.py:751-761` passes the presence array through without context. Matters for synthesized-leave detection and `memberKey` | 3 | | TB4, RTL7h, RTP6e | `attachOnSubscribe`. `ChannelOptions.__init__` (`channeloptions.py:22-26`) takes only `cipher`, `params` and `modes`, and `subscribe()` on the channel, on presence and on annotations all end unconditionally with `await attach()`. This absence also forces the largest single adaptation in the suite, below | 3 | | RSL7 | `RestChannel#setOptions`. The realtime channel implements it; the REST `options` setter expects the kwargs dict `Channels.get` collected, so a `ChannelOptions` raises `TypeError` | 2 | -| RTC1a (TO3h), RTL7f | `echoMessages`, in both the forms RTL7f allows. There is no `echo_messages` client option — passing one raises `TypeError` — and no `echo` connect parameter, so every message the server sends is delivered whatever its `connectionId` | 2 | +| RTL7f | `echoMessages` in its client-filter form. The `echo_messages` option sends `echo=false` (RTC1a), which the server honours, but the client does not itself drop a message carrying its own `connectionId`, and that is the only form a mock-backed test can observe | 1 | | RTP12, RTP12a, RTP12c | `RealtimePresence#history`. The realtime *channel* does delegate `history` to the REST implementation; only the presence object is missing it | 2 | | RTN23c1, RTN23c2 | PING/PONG. `ProtocolMessageAction` stops at `ANNOTATION` (21), so PING (22) and PONG (23) are not modelled and action 22 matches no branch of `on_protocol_message` (`websockettransport.py:37-59`, `:143-199`). The message is counted as activity and discarded | 2 | | TI4, TI1/TI5 | `href` anywhere in the SDK, and `cause` when deserialising. `AblyException.from_dict` and `raise_for_response` read only `message`, `statusCode` and `code`, so both fields are dropped from server errors | 2 | @@ -1198,22 +1439,6 @@ exactly as one arriving while ATTACHED does. **Status:** open bug. -#### `Channels.release` does not detach the channel — 1 test - -**Spec point:** RTS4a. - -Release "detaches the channel and then releases the channel resource". `Channels.release` -(`channel.py:1012-1026`) is `if name not in self.__all: return` followed by -`del self.__all[name]`, and sends nothing. An attached channel is dropped from the -collection while still attached in the Ably service, and the orphaned object stays in -ATTACHED. It overrides the REST implementation, which is correct for REST, without adding -the detach. - -**Tests affected:** `test_rts4a_release_detaches_attached` — `assert 0 == 1` on the -DETACH-message count. - -**Status:** open bug. - #### A decode error other than 40018 has no channel-level handling — 2 tests, 3 cases **Spec point:** PC3. @@ -1790,7 +2015,7 @@ through an internal object to get at a value the specification makes public. | RTP2d1, RTP2h1a, and the `Interface Under Test` blocks of all three presence-map specs | `put(message) -> PresenceMessage?` and `remove(message) -> PresenceMessage?` | both return `bool` (`presencemap.py:111`, `:159`); the message to emit is the caller's own, which `set_presence` appends to `broadcast_messages` when the return is true. `IS NOT null` is read as `is True`. `put` stores a *copy* with the action rewritten to PRESENT and leaves the caller's message untouched, so RTP2d1's "emit the original action" falls out for free | all of `presence_map_test.py` | | RTP19, and the `Interface Under Test` block of `presence_sync.md` | `endSync() -> List`, the synthesized LEAVEs | `end_sync()` returns `(residual, absent)` of the *stored* members; the synthesis lives one level up in `RealtimePresence.set_presence` (`presence.py:575-587`). Tests reading only counts and clientIds concatenate the two lists exactly as `set_presence` does; tests reading the LEAVE itself drive a `RealtimePresence` and assert on what its subscribers receive | 4 in `presence_sync_test.py` | | TB2, RTS3b, RTS3c, RTS3c1, RTL16 | `channel.options` as a `ChannelOptions` | a dict keyed by wire names, because `RealtimeChannel` passes `ChannelOptions.to_dict()` to the REST `Channel` constructor (`channel.py:84`). Assertions read `channel.options['params']['rewind']`. On `ChannelOptions` itself the cipher attribute is spelled `cipher`, not `cipherParams`. `set_options_without_reattach` replaces the stored mapping wholesale rather than merging, which `test_rts3c_options_updated_existing` pins | 5 | -| RTS2, RTS4a | `channels.exists(name)`, `channels.names`, and an awaitable `release()` | `name in client.channels` (`Channels.__contains__`); the collection iterates over its channels rather than their names; `release` is synchronous. Genuinely idiomatic spelling rather than an absence — recorded only because of the `__getattr__` hazard noted below | 4 | +| RTS2, RTS4c, RTS4d, RTS4e | `channels.exists(name)` and `channels.names` | `name in client.channels` (`Channels.__contains__`); the collection iterates over its channels rather than their names. Genuinely idiomatic spelling rather than an absence — recorded only because of the `__getattr__` hazard noted below | 6 | | RSH1b1, RSH1b2, RSH1b3, RSH1b4, RSH1b5, RSH1c3 | `DevicePushDetails`. The specification builds every device as `DeviceDetails(…, push: DevicePushDetails(recipient: {…}))`; ably-python has no such type | `DeviceDetails.__init__` takes `push` as a plain dict and stores it unchanged (`ably/types/device.py:10-40`), and `DeviceDetails.push` hands that dict back, so the tests read `{'recipient': {…}}` directly. The recipient's `transportType` is still validated against `DevicePushTransportType` in the constructor, which is the only part of `DevicePushDetails` carrying behaviour | the 7 in `push_admin_test.py` that register a device, through its `apns_device()` helper | **Status:** open bugs of the missing-API kind, not of the wrong-behaviour kind. Adding the @@ -2173,10 +2398,57 @@ pseudocode is not mistaken for non-compliance. | RTC17 (RSA7b1) | `client.clientId == client.auth.clientId` | `AblyRealtime.client_id` reads `options.client_id` and returns the configured value, while `Auth.__init__` sets `self.__client_id = None` whenever `ably._is_realtime` (`rest/auth.py:34-41`), deferring it to whatever a CONNECTED confirms. The two disagree on a client that has not connected | Open bug. RSA12b only allows the realtime clientId to be unknown while it has not been *configured* | | RTC1f | a `transportParams` boolean appears as `"true"` / `"false"` | `True` / `False`, because `WebSocketTransport.connect` builds the query string with `urllib.parse.urlencode`, which renders each value through `str()` (`websockettransport.py:89`). Integers are unaffected | Open bug. A caller can pass the strings directly, but a bool is what the spec's Stringifiable type admits | +### LiveObjects: the internal shapes the unit specifications assume — S-1 to S-6 + +The objects unit specifications drive internal classes, and ably-python's differ in shape from +the ones they are written against — three of them because the specification contradicts +`objects-features.md` (the first UTS Spec Error entry on the objects tier), three because of how +ably-python is built. Each shape has a label, cited in the module docstring and at the site, so +that one entry here covers every test it affects. The behaviour each test asserts is the +specification's; only the reach into the object differs. + +| Label | The specification | ably-python | Tests | +|---|---|---|---| +| S-1 | `update = obj.applyOperation(...)`, or `result == false` | `apply_operation` returns whether it applied (RTLC7g, RTLM15g) and emits the update through `notify_updated`, no-ops included. `capture_updates(obj)` records them: a specification's `update` is the one update an applied operation emitted, and its `false` is a False return with nothing emitted | the 42 in `internal_live_counter_test.py` and `internal_live_map_test.py` that read an update or a `false` | +| S-2 | The sync and apply state machine on `ObjectsPool`: `processAttached`, `processObjectSync`, `processObjectMessage`, `applyObjectMessages`, `syncState` | On `RealtimeObject`, as in `objects-features.md`: `_on_attached(has_objects)`, `_handle_object_sync_messages(messages, channel_serial)`, `_handle_object_messages`, `_apply_object_messages`, `_sync_state`. A bare `ObjectsPool()` that is then driven is the pool a standalone `RealtimeObject()` holds; `RealtimeObject(pool: pool)` is `RealtimeObject(pool=pool)` | 30 of the 31 in `objects_pool_test.py`, every one but RTO3's; the three RTO5c10 cases in `parent_references_test.py` | +| S-3 | `evaluate(vt)` | `evaluate(vt, timestamp_ms)`: evaluation needs the server time (RTLCV4e, RTLMV4h), which is an async read (RTO16), so the caller fetches it and passes it in. The tests pass a fixed one | the 13 in `value_types_test.py` that evaluate | +| S-4 | The create an evaluated value type retains, as `operation.counterCreate` / `operation.mapCreate` | `operation.resolved_counter_create` / `resolved_map_create`, the `derived_from` of the `*CreateWithObjectId` payload. The two retention tests also assert that `counterCreate` / `mapCreate` is absent from `operation.to_dict()`, which is what "not sent" means and what keeps this reading honest | `test_rtlcv4g5_retains_local_counter_create`, `test_rtlcv4_evaluate_zero_count`, `test_rtlmv4j5_retains_local_map_create`, `test_rtlmv4d_entry_value_types`, `test_rtlmv4d1_nested_value_types`, `test_rtlmv4a_evaluate_validates_entries`, `test_rtlmv4e2_empty_entries`, `test_rtlmv4d_map_set_all_types_table` | +| S-5 | Three reads on the untyped `Instance` that answer null or fail: `value()` on a map (RTINS4d), `size()` on a counter (RTINS9c), and `subscribe` on a primitive failing 92007 (RTINS16c) | The typed instances have no such method (RTTS10a–c; `PrimitiveInstance` has no `subscribe` at all, RTTS7b), and the checked views that do have it refuse the wrapped type with 92007 (RTTS9d). The tests assert the wrapped `type`, that the method is absent, and the 92007 from each view in its place | `test_rtins4_value_counter` and `test_rtins9_size` (their second halves), `test_rtins16c_subscribe_primitive_throws` | +| S-6 | `compact()` on a path or an instance (RTPO13, RTINS10) | ably-python's only snapshot is `compact_json()` (RTTS3f, RTTS7d, LODR-061), which is `compact()` with binary as a base64 string (RTPO14b1) and a map reached again as `{'objectId': id}` (RTPO14b2). The tests take the snapshot through `compact_json()`, and where the specification asserts bytes or the reused dict they assert those two forms, so `compact-cycle-detection-0` asserts what `compact-json-0` does | `test_rtpo13_compact_recursive`, `test_rtpo13c5_compact_cycle_detection`, `test_rtpo13c_compact_counter`, `test_rtpo3c1_read_null_on_failure` (its last line), `test_rtins10_compact` | + +**Status:** shape adaptations, not defects. S-1, S-2 and S-4 close when the specifications are +corrected (candidate O.1); S-3, S-5 and S-6 are how ably-python is built. + +### LiveObjects: inputs that do not exist in Python + +| Test | The specification | The test | Status | +|---|---|---|---| +| `test_rtlmv4a_evaluate_validates_entries` | `LiveMap.create(null)` fails 40003 | `LiveMap.create(entries=None)` is LODR-061's default, so `None` is the omitted argument and evaluates to an empty map, which the test pins. The reachable half of RTLMV4a, a non-dict, is asserted as 40003 | Flips if `create` takes a sentinel default instead; the specification carries no language note here (see O.6) | +| `test_rtlm20_set_invalid_values_table` | A function, `undefined` and a symbol are 40013 | `lambda: None`, `None` and `object()`, each 40013; the parametrize ids keep the specification's labels | JavaScript values given their nearest Python counterparts | +| `test_rtlc12e1_increment_invalid_amounts_table` | The null row applies "where null is distinguishable" | It applies: Python tells `increment(None)` from `increment()`, whose default is 1. `True` is in the table and fails 40003 though `bool` is an `int`; NaN and the infinities are `float('nan')`, `float('inf')` and `float('-inf')` | The specification's own reading, for Python | + +### LiveObjects translation notes + +Not deviations, recorded so the difference from the pseudocode is not mistaken for one. The +largest, the typed views, has its own section at the end of this file. + +| Subject | Note | +|---|---| +| Granted modes | The modes a server grants are bits of the ATTACHED `flags`, so the specifications' `modes: [...]` is `flags=HAS_OBJECTS \| OBJECT_SUBSCRIBE_FLAG`. The standard mock's ATTACHED carries HAS_OBJECTS alone, which leaves `channel.modes` empty, and RTO2 then checks the modes the channel *requested* — which is why every objects channel asks for both through `objects_channel_options()`, and `get()` raises 40024 on one that asked for neither. See the RTO2a row of the features entry above | +| Wire values | The mock tier speaks JSON. A `json` value is a JSON-encoded **string** on the v6 wire (OD2g), so a captured `mapSet.value.json` is compared after `json.loads`; a `bytes` value is base64; an action the specification writes by name compares equal to its `ObjectOperationAction`, an `IntEnum` | +| Numbers | Every number decodes to a `float`, and a locally created value holds one too, so that a value reads the same before and after its echo; a number is read with `value(float)`. A boolean is asserted with `is True` / `is False`, since `True == 1` | +| The public message | `PublicObjectMessage.fromObjectMessage(source, channel)` is `publicmessage.ObjectMessage._from_internal(source, channel_name)`. The public and internal `ObjectMessage` share a name, so the tests reach the public one through the `publicmessage` module | +| Channel state | `channel.object.processChannelState(S)` is `channel.object._act_on_channel_state(ChannelState.S)`, which acts on its argument: RTO23c1's SUSPENDED case and RTO27 call it while the channel is still ATTACHED | +| Loops | RTO17-RTO18's scenario loop and RTO27's DETACHED, FAILED and SUSPENDED blocks are one parametrized test each, keeping one Test ID | +| Listeners | `subscribe` on a path or an instance takes any callable, a bound built-in such as `events.append` included — unlike the channel and connection `EventEmitter` | +| ACK serials | `ack-after-echo-no-double-apply` ACKs `msgSerial` 0, relying, as the specification does, on the first OBJECT of a fresh connection taking it (RTN7b) | +| Integration | An un-awaited `channel.attach()` or write is a task, awaited later inside `pytest.raises` where the specification expects it to fail. `poll_until_success` treats only an `AblyException` as "not yet", so that any other error fails at once. `channel.object.get()`, which the lifecycle and sync specifications leave unbounded, is bounded at the fifteen seconds `objects_gc_test.md` gives it | + ## Mock Infrastructure Limitations Tests that cannot be implemented as written, kept as skipped stubs carrying their Test -IDs so the specification's coverage is still accounted for. Fifteen in total. Two of the +IDs so the specification's coverage is still accounted for. Fifteen in total, none of +them from `uts/objects`. Two of the entries are caused by the SDK rather than by the mock, but they land here because the effect is the same: no test can observe the behaviour. @@ -2384,6 +2656,81 @@ is not shared between them. duplicates, on both protocols. The test asserts the full set rather than only the specification's `>= 5`, since the fixture is fixed. +### The sandbox delivers the echo before the ACK, so RTO9a3 is reached at the mock tier only + +Against the sandbox the echo of a client's own write arrived before its ACK in every trace, +the first write included, where both frames came in one read with the OBJECT first. The echo +is therefore applied as a CHANNEL operation (RTO8b), the ACK's LOCAL apply is rejected by the +RTLO4a site-serial check, and `_applied_on_ack_serials` (RTO9a3) is never written on that path. +Both orders are correct per the specification, and nothing is applied twice — a probe writing +every value type in every protocol combination found the writer's state unchanged by its +echoes. The ACK-first order, where RTO9a3 does the work, is covered at the mock tier, whose +standard mock ACKs before any echo is injected (`test_rto20_echo_dedup`, and RTO9a3 in +`objects_pool_test.py`). + +The same traces show the sandbox pacing ACKs at about one per 500 ms per connection: the first +ACK arrives with the echo, and later ones on a half-second tick. An awaited write after the +first can take that long without anything being wrong, which is why the GC integration tests +take over a second each. + +### A Python-formatted `initialValue` is accepted, because the server hashes the string it receives + +An object id is a hash over `[initialValue]:[nonce]` (RTO14b1), and ably-python formats +`initialValue` as Python's `json.dumps` does — `{"count":42.0}` where ably-js writes +`{"count":42}`, and non-ASCII as `\uXXXX` escapes where ably-js writes raw UTF-8 — so it looked +as though the server, or another SDK, would compute a different id. Measured against the +sandbox's REST objects endpoint with client-generated ids: Python's formatting, ably-js's +formatting and raw UTF-8 are each accepted (201) when the id is hashed over the string sent, +while an id hashed over `{"count":7.0}` and sent with `{"count":7}` is rejected 400/92000, +"object id does not match expected value". The server hashes the exact string it receives. The +evaluation computes the id from the one string it then sends, so the formatting cannot diverge, +and the integration tier confirms it over realtime in both protocols. + +### A sync cursor can itself contain `:`, and splitting on the first one is right + +A sync the sandbox split into twenty OBJECT_SYNC messages carried channelSerials of the shape +`4d58ad59:map:0TeL…@1791551583365` and ended on `4d58ad59:`, so the cursor contains `:` too. +RTO5a1 splits on the first `:`, which yields the right sequence id; late joiners on both +protocols matched the writer exactly. + +### `RTO5a2-RTO17/sync-interrupted-reconnect-0` does interrupt a sync + +Its two siblings in `objects_faults.md` never reach what they are named for (see the objects +entry under UTS Spec Errors), so this one was measured too, and it does: the proxy drops the +first OBJECT_SYNC and closes the socket, the client resumes on the same connectionId, receives +an ATTACHED carrying RESUMED and HAS_OBJECTS and a fresh OBJECT_SYNC, and `get()` resolves from +that. The ATTACHED arrives on a channel already ATTACHED, which emits no state change, so the +re-sync depends on the channel handing every ATTACHED it accepts to the objects — which it does. + +### Where ably-python departs from ably-js, it follows `objects-features.md` + +Recorded because anyone comparing the two implementations will meet these first. + +| Spec point | ably-js | ably-python | +|---|---|---| +| RTO5c2, RTO5c10 | `_applySync` returns early when the SyncObjectsPool is empty, skipping both the removal of absent objects and the parent-reference rebuild | No early return; both run, as written | +| RTO5e | Moves to SYNCING only on a new sequence, so an OBJECT_SYNC with no sequence id while SYNCED emits nothing | Applies RTO5e to every OBJECT_SYNC, emitting SYNCING then SYNCED | +| RTO5a6 | `^([\w-]+):(.*)$`, so `:x` reads as having no channelSerial | Malformed means "does not contain the `:` separator", as written | +| RTO2 | Requires a granted mode to be among the requested ones as well | RTO2b applies only "otherwise", when no modes were granted | +| RTLC9 | A COUNTER_INC with no `counterInc` is not applied | The RTLC9h no-op, applied, reading RTLC7d5's "passing in `operation.counterInc`" as written | +| RTLMV4c | A non-finite number in a map's entries is sent as `null` | 40013: it has no JSON representation (OD4d3), and `initialValue` is always JSON | + +### More `Task was destroyed` warnings, and a GC timer on every attached channel, are not faults + +A plain run of `test/uts` prints `Task was destroyed but it is pending!` 174 times where the +baseline printed 80. Every one is a `ConnectionManager.connect_base()` task, the teardown +warning the baseline prints already, and the rise comes from the objects cases that connect a +mock-backed client; no new kind of warning appears. + +The GC timer is scheduled on the first ATTACHED of every channel, whether or not it uses +objects (RTO10a; ably-js does the same whenever the plugin is loaded), and cancelled on +DETACHED and FAILED, so a client's `close()` stops it. A non-objects test advancing a +`FakeClock` by five minutes therefore runs one empty sweep. `Channels.release()` stops it +too, although release does not detach — the RTS4a defect under *Failing Tests*. And a burst +of creates on a channel that has +not yet read the server time each request `/time`, until the first answer persists the offset +(RTO16a); wasteful, and not wrong. + ## Candidate issues `writing-derived-tests.md` asks for the deviations above to be classified into distinct @@ -2401,7 +2748,7 @@ is missing accessors over correct behaviour. Within a tier the order is blast ra | Issue | Covers | Rows below it answers | |---|---|---| | [#706](https://github.com/ably/ably-python/issues/706) | the fabricated `"None:0"` message id | 3.3 | -| [#658](https://github.com/ably/ably-python/issues/658) | presence messages sent on reconnection before reattach | adjacent to 3.14, which is the other half of RTP17 automatic re-entry | +| [#658](https://github.com/ably/ably-python/issues/658) | presence messages sent on reconnection before reattach | adjacent to 3.13, which is the other half of RTP17 automatic re-entry | | [#656](https://github.com/ably/ably-python/issues/656) | `utcfromtimestamp` deprecation | adjacent to 2.4 | | [#709](https://github.com/ably/ably-python/issues/709)–[#712](https://github.com/ably/ably-python/issues/712) | REST request timeout, token-request nonce reuse, single-host retry, `dispose()` teardown | adjacent to I.3, which is a second defect on the line #709 is about and is not covered by it; otherwise none, these having been filed from the REST derivation | @@ -2636,33 +2983,27 @@ disconnect per RTN7e", which issue 1.2 above proves false. No gated test: the UT this cannot distinguish the behaviours (recorded under UTS Spec Errors), so the finding is the output. -**3.9 `Channels.release` does not detach the channel.** RTS4a. `channel.py:1012-1026` deletes -the entry and sends nothing, so the channel is dropped from the collection while still -attached in the Ably service — the application goes on being billed for and delivered to a -channel it believes it released. -`test/uts/realtime/unit/channels/channels_collection_test.py -k rts4a_release_detaches_attached` - -**3.10 The UPDATE event drops the CONNECTED message's error.** RTN24. `on_connected` +**3.9 The UPDATE event drops the CONNECTED message's error.** RTN24. `on_connected` (`connectionmanager.py:425-428`) builds the `ConnectionStateChange` without the `reason` it was handed, so an application never sees the error a CONNECTED carries — including the RTN15c7 failed-resume error, which RTN25 lists among those that must set `Connection#errorReason`. A one-line fix. `test/uts/realtime/unit/connection/update_events_test.py -k rtn24_update_event_with_error` -**3.11 Detach protocol faults.** RTL5i — a detach requested while already DETACHING sends a +**3.10 Detach protocol faults.** RTL5i — a detach requested while already DETACHING sends a **second** DETACH, because `_request_state` calls `_check_pending_state()` even after `_notify_state` returns early. RTL5k — an ATTACHED arriving while DETACHING or DETACHED is ignored instead of answered with a new DETACH, so the detach times out and the channel returns to ATTACHED. Same method, one issue. `test/uts/realtime/unit/channels/channel_detach_test.py -k "rtl5i or rtl5k"` -**3.12 The deleted RTL4j ATTACH_RESUME flag is still set on every reattach.** RTL4j, deleted +**3.11 The deleted RTL4j ATTACH_RESUME flag is still set on every reattach.** RTL4j, deleted in specification 6.1.0. `_notify_state` sets `__attach_resume` on every ATTACHED and `_encode_flags` ORs it in, so every reattach carries `flags: 32` and tells the server something the server now decides for itself. `test/uts/realtime/unit/channels/channel_attach_test.py -k rtl4j` -**3.13 A decode error other than 40018 never fails the channel.** PC3. `channel.py:744-748` +**3.12 A decode error other than 40018 never fails the channel.** PC3. `channel.py:744-748` gives channel-level handling to 40018 alone; every other decode error is logged and the batch **silently skipped**, with no state change and no `error_reason`, so a vcdiff message with no decoder is simply lost. Compounding it, `message.py:302-305` checks the delta's `from` id @@ -2670,34 +3011,34 @@ decoder is simply lost. Compounding it, `message.py:302-305` checks the delta's missing-decoder branch at all. `test/uts/realtime/unit/channels/channel_delta_decoding_test.py -k pc3_no_plugin_fails` -**3.14 A failed automatic re-entry reports the NACK, not the 91004 wrapper.** RTP17e. +**3.13 A failed automatic re-entry reports the NACK, not the 91004 wrapper.** RTP17e. `_reenter_member` (`presence.py:667-674`) emits the UPDATE with `resumed=False` and the raw NACK as `reason`, where the spec requires `resumed` true and a 91004 naming the clientId with the NACK as `cause`. Local to one method. `test/uts/realtime/unit/presence/realtime_presence_reentry_test.py -k rtp17e` -**3.15 `ping()` rejects DISCONNECTED, and charges the connect wait to the caller's timeout.** +**3.14 `ping()` rejects DISCONNECTED, and charges the connect wait to the caller's timeout.** RTN13b, RTN13c, RTN13d. `ConnectionManager.ping` (`:362`) admits only CONNECTED and CONNECTING where RTN13d requires a ping from DISCONNECTED to be deferred; and `:375` enters `asyncio.wait_for` as soon as `ping()` is called, so a ping requested while CONNECTING can expire before its HEARTBEAT has gone out. `test/uts/realtime/unit/connection/connection_ping_test.py -k "rtn13b_deferred or rtn13c or rtn13d"` -**3.16 A failed RTN22 reauth leaves no trace on the connection.** RSA4c1, RSA4c3. +**3.15 A failed RTN22 reauth leaves no trace on the connection.** RSA4c1, RSA4c3. `websockettransport.py:170-175` awaits `auth.authorize()` inside a bare `except Exception` that only logs, so nothing reaches `on_error_from_authorize` and `errorReason` stays as it was. **Hold this one**: specification#466 would make the current behaviour correct, and the two UTS specs disagree about it today. `test/uts/realtime/unit/auth/connection_auth_test.py -k rsa4c3_callback_error_stays_connected` -**3.17 TokenParams passed to an authCallback carry no clientId on a realtime client.** +**3.16 TokenParams passed to an authCallback carry no clientId on a realtime client.** RSA12a, RTN2e. `Auth.__init__` nulls `client_id` for a realtime client (`rest/auth.py:36-41`) and `_ensure_valid_auth_credentials` only adds it when non-null, so an auth server never learns the configured clientId. The REST client does pass it. The same nulling is why `client.client_id` and `client.auth.client_id` disagree before CONNECTED (RTC17). `test/uts/realtime/unit/auth/connection_auth_test.py -k rtn2e` -**3.18 No 40171 log at instantiation with a non-renewable token.** RSA4a1. Requires an +**3.17 No 40171 log at instantiation with a non-renewable token.** RSA4a1. Requires an info-level record carrying 40171 and the TI5 help URL; the SDK logs at debug with no code, and `grep -rn href ably/` finds no help URLs at all. Small, and the TI5 half is a gap of its own. @@ -2715,7 +3056,7 @@ Each row is one feature and one issue. None is a bug in existing code. | Retry backoff, jitter and `retryIn` on both state-change types | RTB1, RTB1a, RTB1b | 4 | `connection/backoff_jitter_test.py` | | `RealtimeChannel#whenState` (the connection has a private equivalent) | RTL25, RTL25a, RTL25b | 4 | `channels/channel_when_state_test.py` | | `attachOnSubscribe` on `ChannelOptions`. Also forces the suite's largest adaptation — 21 tests attach explicitly to work around it | TB4, RTL7h, RTP6e | 3 | `channels/channel_subscribe_test.py -k rtl7h`, `channels/channel_options_test.py -k tb4`, `presence/realtime_presence_subscribe_test.py -k rtp6e` | -| `echoMessages`, in both the client-filter and `echo`-parameter forms | RTC1a, RTL7f | 2 | `client/realtime_client_test.py -k rtc1a`, `channels/channel_subscribe_test.py -k rtl7f` | +| `echoMessages` in its client-filter form; the `echo` connect parameter is sent | RTL7f | 1 | `channels/channel_subscribe_test.py -k rtl7f` | | `RealtimePresence#history` (the realtime *channel* does delegate `history`) | RTP12, RTP12a, RTP12c | 2 | `presence/realtime_presence_history_test.py` | | PING/PONG handling — actions 22 and 23 are not modelled | RTN23c1, RTN23c2 | 2 | `connection/heartbeat_test.py -k rtn23c1` | | The `heartbeats` connect parameter. Binding on ably-python, which cannot observe ping frames | RTN23a | 1 | `connection/heartbeat_test.py -k rtn23a_heartbeats_true` | @@ -2819,6 +3160,87 @@ the unit tier adapts to that; RTN14b does not, which is what makes this a defect than a choice. `test/uts/realtime/integration/proxy/connection_open_failures_test.py -k rtn14b` +### From the objects derivation + +These are against `ably/specification`, not ably-python: the objects tier found no SDK +non-compliance, and each item is a fault in `uts/objects` or, for O.7, a gap in +`objects-features.md`. **None is filed.** Each matches an entry under *UTS Spec Errors*, which +carries the detail, and the command after it runs the derived tests that correct or work +around it — they pass, and are the reference for what the specification should say; +`RUN_DEVIATIONS` makes no difference to them. They are ranked by what an SDK deriving from the +specification as written would suffer: first where a conforming SDK fails or has to adapt +(O.1–O.3), then where a non-conforming one passes (O.4, O.5), then housekeeping (O.6), with the +features-spec gaps last, being questions for a different document. + +**O.1 The unit specifications model three internal interfaces against `objects-features.md`.** +RTLC7g, RTLM15g, RTO3a, RTO7, RTO17, RTLCV4g5, RTLMV4j5, MCRO2, CCRO2. +`internal_live_counter.md` and `internal_live_map.md` take an update from `applyOperation` +while asserting, of the same call, the boolean RTLC7g and RTLM15g say it returns; +`objects_pool.md` puts the sync state machine on `ObjectsPool`, and `PLAN.md` repeats it; +`value_types.md` and `public_object_message.md` read the retained create in two different +places, one of which would put it on the wire, and the latter gives `*CreateWithObjectId` fields +MCRO2 and CCRO2 do not define. Every SDK has to adapt all of it — S-1, S-2 and S-4 here, and +ably-js's derived suite likewise. One issue: write the files against the features spec's shapes. +`test/uts/objects/unit/internal_live_map_test.py test/uts/objects/unit/objects_pool_test.py test/uts/objects/unit/value_types_test.py test/uts/objects/unit/public_object_message_test.py` + +**O.2 Fixtures put values where the protocol does not.** RTO2, RTO25a, RTO26a, RTLMV4d, +RTLMV4f1. Granted modes written as an ATTACHED `modes: [...]` field rather than as `flags` bits; +`connectionId` inside `connectionDetails` in the helper specification and two hand-written +mocks; RTLMV4d's bytes row comparing an in-memory entry with its base64 wire form; and the +provisioning comment's one result per batch entry, where the server answers one result for +the batch. A conforming SDK fails the first three as written. +`test/uts/objects/unit/realtime_object_test.py test/uts/objects/unit/value_types_test.py -k "rto2_mode or rto25a or rto26a or rtlmv4d_map_set_all_types_table"` + +**O.3 Steps read state as though an injected frame were applied at once.** RTO5c9, RTO10c1b1, +RTO17, RTO18d, RTO19, RTO20, RTO20d4, RTO20e, RTO20e1, RTO23c1, RTO24b1, RTO24c1, RTINS16, +RTINS16h, RTLO4b, RTPO19, and at the integration tier RTO5a2, RTO7, RTO8, RTO17. An SDK that +processes frames on a task of its own **fails** the three RTO23c1 tests, the two RTO20e1 tests, +RTO20e and RTO20d4's empty-list test as written — the operation starts before the re-sync +ATTACHED has been processed, sees SYNCED and resolves — while the negatives of RTO20's +echo-dedup, RTO19, RTO10c1b1 and RTINS16h pass whatever the SDK does, and every exact count +after a `poll_until(>= n)` is half a negative. `path_object.md` already waits for an inbound +frame in its compact tests and `standard_test_pool.md` already defines the quiescence pattern; +the fix is to use them throughout, with `process_pending_events()` after each frame that is +read from, and, in `objects_faults.md`, a recorded state sequence in place of `AWAIT_STATE` for +DISCONNECTED and for a re-attach. +`test/uts/objects/unit/realtime_object_test.py test/uts/objects/unit/instance_test.py -k "rto23c1 or rto20e or rto20d4_empty or rto20_echo or rto19 or rto10c1b1 or rto5c9_rto20 or rtins16h"` +`test/uts/objects/integration/proxy/objects_faults_test.py` + +**O.4 Tests that pass without the behaviour they are named for.** RTLM8g, RTLM9b, RTO10, +RTO10b1, RTO10c1b1, RTPO6b, RTPO19 with RTO24b2a2, RTO4b2, RTO14b1, RTINS16g, RTO20e, RTO23c, +RTO20c, RTO20d1, RTO20e1; and, derived as written, RTO4 with RTO5a2, RTO7 and RTO8 through the +proxy, RTLC11b1, RTLO4b, RTO5c10 and the lifecycle test's unread `events_b`. Sixteen derived +tests carry the assertion or the direct check their specification omits. The sharpest are the GC +tests, which read a value that is null with no GC at all — RTO10b1 also advances less than one +GC interval — and the two `objects_faults.md` tests titled for RTO8a's buffer, neither of which, by +measurement, exercises the buffer. Same class as +[#543](https://github.com/ably/specification/issues/543). +`test/uts/objects/unit -k "rtlm8g or rtlm9b or rto10 or rtpo6_at_escaped or rtpo19_map_clear or rto4b_attached or rto14_objectid_format_counter or rtins16g or rto20e or rto23c_get_waits or rto20c_missing or rto20d1"` + +**O.5 Boundaries no test reaches.** RTLM7h, RTLM8g, RTLM24c, RTLM19a1, RTLC7c, RTLM15c, +RTLC6a, RTLM6a, RTLM7a3, RTLC7b, RTLM15b, RTLM23c, RTLO6b, RTINS3b. Equal serials against the +clear floor, an entry exactly one grace period old, the order of serial bookkeeping against the +tombstone checks, a rejected MAP_SET's parent references, the serials a rejected operation +leaves, MAP_CREATE no-ops, the clock a map-entry tombstone reads, and a primitive instance's +`id`. Each but the last was found as a mutant every existing test lets through, so each new +test can be written against that mutant. No reproduction: there is no test to run. + +**O.6 Labels, commentary and harmless fixture slips.** The thirteen rows of the objects +housekeeping entry: RTLM24's `<=`, RTLM14c's title, RTO20d4's commentary, an RTO19 test that +exercises RTO18f2, RTO20's credit to RTO9a3, RTPO6b's undefined 40003, RTLM15e's half-set +tombstone, the second `root` beside the pool's, the ATTACHED `channelSerial`s, fragile bare +serials, two missing language notes, and `integration-testing.md`'s Protocol Variants list. +None changes an outcome. Same class as +[#532](https://github.com/ably/specification/issues/532). + +**O.7 `objects-features.md` gaps.** RTLM5d2, OD4c5, OD4d5, RTO4, RTO5a5, RTO5a6, RTO5d, RTO4b2, +RTO14b2, RTO2a, RTPO6; and, on the unmerged pull requests, RTLMV4c1 +([#491](https://github.com/ably/specification/pull/491)) and RTPO20c and RTBC16d +([#471](https://github.com/ably/specification/pull/471)). The widest reaching are RTLM5d2, +under which every JSON map entry reads null, and RTO5d, under which one ObjectMessage with no +`object` stalls the whole sync. The two #471 points want raising on that pull request before it +merges, since ably-js already disagrees with its text on the first. + ## How the specifications are adopted here Choices about the approach, as against the behaviour recorded above. @@ -3172,6 +3594,104 @@ and each message still gets a distinct id, so only the setup latency differs. Ea then polls history until the expected count is visible before paginating, which is the specifications' own `poll_until`, because history is not immediately consistent. +### The objects tier mirrors the other two, with a pure tier inside its unit tier + +`uts/objects/unit/.md` becomes `test/uts/objects/unit/_test.py`, +`uts/objects/integration/_test.md` becomes `test/uts/objects/integration/_test.py`, +and `integration/proxy/objects_faults.md` becomes `integration/proxy/objects_faults_test.py`, +whose Test IDs read `objects/proxy/...`. `uts/objects/helpers/standard_test_pool.md` becomes +`test/uts/objects/helpers/standard_test_pool.py`, with tests of its own beside it, which are +the 18 harness cases the header counts separately. + +Seven of the fifteen unit specifications — `internal_live_counter`, `internal_live_map`, +`objects_pool`, `parent_references`, `object_id`, `value_types` and `public_object_message`, +161 Test IDs — construct internal objects and connect nothing. The other eight, 161 Test IDs +more, drive `channel.object` over the mock websocket. The specifications draw no such line, +but it decides which shapes apply: S-1 to S-4 are pure-tier adaptations, and the mock tier +reaches the library only through its public surface and a handful of underscore readers. + +### The specifications' untyped `PathObject` and `Instance` become LODR-061's typed views + +The specifications are written against the merged features spec's `PathObject` and +`Instance`, each one class carrying every method. ably-python follows LODR-061 and the RTTS +partition of [#491](https://github.com/ably/specification/pull/491): navigation, `instance`, +`compact_json` and `subscribe` are on the base `PathObject`, and a type-specific +method is reached through the view for that type. So `root.get("score").value()` is +`root.get('score').as_live_counter().value()`, `root.get("name").value()` is +`root.get('name').as_primitive().value()`, and `pathObject.set(k, v)` is +`await path_object.as_live_map().set(k, v)`; `root` is already a `LiveMapPathObject` and +needs no view. A path's views are unchecked — a view of the wrong type reads `None`, or `[]` +for a collection (RTTS5d1), and a write through it raises 92007 or 92005 (RTTS5d2) — so +where a specification expects the untyped `value()` to be null, the test asserts it through +both value views. An `Instance`'s views are checked and raise 92007 (RTTS9d). + +That is a difference of spelling, which `writing-derived-tests.md` says is not a deviation, +and nothing above records it test by test. Where the partition makes a read unreachable +rather than differently spelled, the test is an S-5 adaptation, and where the specification +calls `compact()`, which the partition leaves out (RTTS3f, RTTS7d), it is an S-6 one. +Mutations are `async` and awaited; reads, navigation, the views and `subscribe` are +synchronous; `entries()` yields `(key, value)` tuples; `Instance.id` and `Instance.type` are +properties. + +### A standalone `RealtimeObject` drives the pure tier + +`objects-features.md` puts the sync state machine on `RealtimeObject` (S-2), so the pure +tier needs one without a channel. `RealtimeObject()` built with none, or +`RealtimeObject(pool=pool)` around a pool a test filled, is a working state machine: `get()` +waits for SYNCED and returns the root, `_publish` raises 40000, the server time is the local +clock, the channel-mode and channel-state preconditions check nothing, and a path +subscription's events carry no channel name. None of that is reachable through the public +API; it exists so that a pure test exercises the code the mock tier runs. Members of a +public class that LODR-061 does not name carry a leading underscore — `_objects_pool`, +`_sync_state`, `_on_attached`, `_gc_interval_ms` — and the tests reach them by those names. + +### The objects mock tier speaks JSON, and answers `/time` + +`setup_synced_channel` connects through `objects_client`, which sets +`use_binary_protocol=False`, so the builders' dictionaries go over the mock exactly as the +specification describes the JSON wire: camelCase keys, numeric actions and semantics, a `json` +value as a JSON-encoded string and `bytes` as base64. The msgpack wire is covered by the +integration tier, which runs its three specifications under both protocols, and by +`standard_test_pool_test.py`, which round-trips every wire type through both encodings. + +Creating a `LiveCounter` or `LiveMap` reads the server time (RTO16), which is a REST request, +so `objects_client` also installs a `MockHttpClient` answering `GET /time` from the test's +clock and 404 to anything else. Without it a mock-backed client would reach the network for +the time. The harness CONNECTED carries a `siteCode`, an `objectsGCGracePeriod` and a +`maxIdleInterval` of 0, so that a `FakeClock` test — the GC tests, chiefly — has no idle timer +to fire. + +### The objects integration tier provisions its own sandbox app, and its proxy clients sign a JWT + +`test/uts/objects/integration/conftest.py` provisions a sandbox app for the tier, separate +from the realtime tier's, whose fixtures are not visible from `test/uts/objects`; it is +called `realtime_sandbox` all the same, so the tests read as the realtime tier's do. All three +specifications outside `proxy` carry Protocol Variants, so every test there takes the +`use_binary_protocol` fixture. Objects are provisioned over REST with `X-Ably-Version: 6`, in +JSON whichever protocol the realtime client speaks. The proxy package repeats the realtime +tier's `proxy_control` and `proxy_session` fixtures and its 300-second marker, prepended so +that it wins over the parent's 120. A client under test there authenticates with an +`auth_callback` returning a locally signed Ably JWT, which costs no round trip and so puts +nothing in the event log beside the frames a test counts; a second client, where a test has +one, goes straight to the sandbox with the key, as the specification builds it. + +### Batch and the typed views are tested outside `test/uts`, for want of a specification + +No UTS specification covers batched writes (RTPO20, RTINS17, RTBC1–RTBC16, in the unmerged +[#471](https://github.com/ably/specification/pull/471)) or the typed views (RTTS2–RTTS10, in +#491). Both are implemented, so both are tested, in `test/unit/liveobjects/`: `batch_test.py`, 29 +functions run as 76 cases, and `typed_views_test.py`, 17 functions run as 61. They are written +against `objects-features.md`, the two pull requests and LODR-061, carry no `# UTS:` comment, +and are not counted in the header. Beside them, `realtime_object_test.py`, +`subscription_test.py` and `value_types_test.py` (17 functions, 18 cases) hold regression +tests for behaviour no specification test reaches: a sync wait on a channel that is already +detached, failed or released, an object message that fails to decode, a GC sweep that raises, +and the validation of a creation value before the server time is read. They drive the standard synced pool over the mock +websocket with this tier's helpers, and their own `conftest.py` closes clients as +`test/uts/conftest.py` does. Single-point mutants of the implementation checked that they +bite: they kill 15 of 16 in the batch code — the survivor removes a check another layer +repeats — and all 7 in the view code. + ### Deviation records are consolidated, not accumulated Each round of derivation runs a specification area per agent, and each writes its own @@ -3187,7 +3707,11 @@ new ones: `Auth#revokeTokens`, `Rest#batchPresence`, the `PushChannel` surface a same root cause as RTN15h1's, and sits in that entry. The realtime integration round did the same for five more: the RTN15h3 stall, the connection-level ERROR bypass, the unused `connectionStateTtl`, connection recovery, and the missing channel-level handling for a -decode error other than 40018. +decode error other than 40018. The objects round found its specification faults several +times over in the same way: the internal shapes the unit specifications assume were +reported by three agents, the missing barriers after an injected frame by four, and the +RTO8a buffer that `objects_faults.md` never exercises by the agent that wrote the contract, +the one that derived the tier and the one that measured it. A verdict can change the same way, when a second specification reaches a behaviour the first was content with. `errorReason` surviving a successful reconnect is permitted by @@ -3207,16 +3731,22 @@ The header states how many derived tests there are, how many pass, how many are gated and how many cannot run. Those numbers are the check that the file is still true: in pytest cases, the gated count must equal the number of failures under `RUN_DEVIATIONS=1`, and gated plus unrunnable must equal the number of skips without -it. As of this writing that is 217 failures and 15 skips with the variable set, and -232 skips and 1124 passes without it, the 1124 being 1002 derived cases and 122 -`helpers/` ones. +it. As of this writing that is 170 failures, 1571 passes and 15 skips with the variable +set, and 185 skips and 1571 passes without it, the 1571 being 1423 derived cases and 148 +harness ones — 130 under `helpers/` and 18 under `objects/helpers/`. Those figures need +the `submodules` checkout CI makes: without it, three `rest/unit/encoding` tests that read +the ably-common fixtures skip as well. The tiers are told apart by the junit report's +`classname`, `test.uts..`, with `proxy` the package within an integration tier. The other two counts are measured from the source rather than from a run. The number of -**derived tests** is the number of `# UTS:` comments, 1141. The number of **Test IDs** is -the number of *distinct* ids in them, 1132 — not the same figure, because five ids in +**derived tests** is the number of `# UTS:` comments, 1481. The number of **Test IDs** is +the number of *distinct* ids in them, 1472 — not the same figure, because five ids in `rest/unit` are carried by more than one test function. Counting the comments and calling the result Test IDs is the easy mistake here, and it overstates the specification coverage -by nine. +by nine. The other is to gate or un-gate a test, record it in the body, and leave the +header alone. A test moved from passing to `@spec_error` moves the passing, gated, +per-side and specification-fault figures together, and this file's were a case short for a +while after `test_rsc7d_ably_agent_header_format` was gated that way. Anyone changing the suite should re-run both and update the header, rather than copying the previous numbers forward. Keep the three units apart while doing it: one Test ID is diff --git a/test/uts/helpers/client.py b/test/uts/helpers/client.py index 57db6e6a..061a6f6e 100644 --- a/test/uts/helpers/client.py +++ b/test/uts/helpers/client.py @@ -4,8 +4,8 @@ import inspect import logging -from ably import AblyRealtime, AblyRest -from ably.types.testoptions import TestOptions +from ably.pubsub.server import create_http_client, create_realtime_client +from ably.pubsub.types.testoptions import TestOptions from test.uts.helpers.clock import settle from test.uts.helpers.sandbox import SANDBOX_ENDPOINT @@ -41,7 +41,7 @@ def rest_client(mock_http, clock=None, **kwargs): if not any(option in kwargs for option in CREDENTIAL_OPTIONS): kwargs['key'] = DEFAULT_KEY test_options = TestOptions(http_transport=mock_http.as_transport(), clock=clock) - client = AblyRest(_test_options=test_options, **kwargs) + client = create_http_client(_test_options=test_options, **kwargs) __open_clients.append(client) return client @@ -63,7 +63,7 @@ def realtime_client(mock_websocket=None, mock_http=None, clock=None, **kwargs): kwargs['key'] = DEFAULT_KEY kwargs.setdefault('auto_connect', False) kwargs.setdefault('fallback_hosts', []) - client = AblyRealtime(_test_options=TestOptions( + client = create_realtime_client(_test_options=TestOptions( http_transport=mock_http.as_transport() if mock_http is not None else None, websocket_connect=mock_websocket.as_connect() if mock_websocket is not None else None, clock=clock, @@ -92,7 +92,7 @@ def sandbox_rest_client(key=None, **kwargs): kwargs['key'] = key kwargs.setdefault('endpoint', SANDBOX_ENDPOINT) kwargs.setdefault('use_binary_protocol', False) - client = AblyRest(**kwargs) + client = create_http_client(**kwargs) __open_clients.append(client) return client @@ -112,7 +112,7 @@ def sandbox_realtime_client(key=None, **kwargs): kwargs['key'] = key kwargs.setdefault('endpoint', SANDBOX_ENDPOINT) kwargs.setdefault('use_binary_protocol', False) - client = AblyRealtime(**kwargs) + client = create_realtime_client(**kwargs) __open_clients.append(client) return client @@ -274,7 +274,7 @@ async def connected_client(mock_websocket, **kwargs): Most channel specifications open this way, since a channel cannot attach until the connection carrying it is up. """ - from ably.realtime.connection import ConnectionState + from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.mock_websocket import CONNECTED_MESSAGE if mock_websocket.on_connection_attempt is None: @@ -293,7 +293,7 @@ async def drop_transport(client, mock_websocket): specification expects to find it rather than reconnecting behind the assertions. Returns the connection states recorded along the way. """ - from ably.realtime.connection import ConnectionState + from ably.pubsub.realtime.connection import ConnectionState states = [] @@ -316,7 +316,7 @@ async def reconnect_transport(client, mock_websocket, connected_message=None): Waiting on the connection state alone would be satisfied by the CONNECTED the client already holds, so this counts a fresh arrival. """ - from ably.realtime.connection import ConnectionState + from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.mock_websocket import CONNECTED_MESSAGE message = CONNECTED_MESSAGE if connected_message is None else connected_message diff --git a/test/uts/helpers/clock.py b/test/uts/helpers/clock.py index 6d818715..c2a07720 100644 --- a/test/uts/helpers/clock.py +++ b/test/uts/helpers/clock.py @@ -61,7 +61,7 @@ def __repr__(self): class FakeClock: """Records timers against a notional time which only `advance` moves. - `FakeClock` has the shape of `ably.util.clock.Clock`, so it can be passed + `FakeClock` has the shape of `ably.pubsub.util.clock.Clock`, so it can be passed straight to `TestOptions(clock=...)`. Nothing fires until a test asks for it, and every reading the client takes moves only when `advance` does. """ diff --git a/test/uts/helpers/mock_http_test.py b/test/uts/helpers/mock_http_test.py index d860a7d0..e48d65a8 100644 --- a/test/uts/helpers/mock_http_test.py +++ b/test/uts/helpers/mock_http_test.py @@ -6,9 +6,9 @@ import msgpack import pytest -from ably import AblyRest -from ably.types.testoptions import TestOptions -from ably.util.exceptions import AblyException +from ably.pubsub.server import create_http_client +from ably.pubsub.types.testoptions import TestOptions +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.mock_http import MockHttpClient KEY = 'a.b:c' @@ -20,7 +20,7 @@ def rest_client(mock, **kwargs): """A REST client whose HTTP calls are served by `mock`.""" - return AblyRest(key=KEY, _test_options=TestOptions(http_transport=mock.as_transport()), **kwargs) + return create_http_client(key=KEY, _test_options=TestOptions(http_transport=mock.as_transport()), **kwargs) async def raw_get(ably, path='/time'): @@ -175,7 +175,8 @@ def connect(connection): mock = MockHttpClient(on_connection_attempt=connect, on_request=lambda request: request.respond_with(200, [SERVER_TIME])) - ably = AblyRest(token='foo', tls=False, _test_options=TestOptions(http_transport=mock.as_transport())) + ably = create_http_client(token='foo', tls=False, + _test_options=TestOptions(http_transport=mock.as_transport())) await ably.time() diff --git a/test/uts/helpers/mock_websocket.py b/test/uts/helpers/mock_websocket.py index bbf0075a..fc89bb93 100644 --- a/test/uts/helpers/mock_websocket.py +++ b/test/uts/helpers/mock_websocket.py @@ -27,7 +27,7 @@ import msgpack from websockets.exceptions import WebSocketException -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.mock_http import RecordedUrl MSGPACK_PROTOCOL = 'application/x-msgpack' diff --git a/test/uts/helpers/mock_websocket_test.py b/test/uts/helpers/mock_websocket_test.py index dc2866f2..6ecb74a3 100644 --- a/test/uts/helpers/mock_websocket_test.py +++ b/test/uts/helpers/mock_websocket_test.py @@ -12,8 +12,8 @@ import msgpack import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import FakeClock from test.uts.helpers.mock_http import MockHttpClient @@ -189,7 +189,7 @@ async def test_a_pending_connection_carries_the_headers_the_client_sent(): headers = mock.connection_attempts[0].headers assert 'ably-agent' in headers - assert 'ably-python' in headers['Ably-Agent'] + assert 'ably-pubsub-python' in headers['Ably-Agent'] async def test_the_protocol_follows_use_binary_protocol(): diff --git a/test/uts/helpers/presence.py b/test/uts/helpers/presence.py index 4d493bdc..cec057a9 100644 --- a/test/uts/helpers/presence.py +++ b/test/uts/helpers/presence.py @@ -8,10 +8,10 @@ from types import SimpleNamespace -from ably.realtime.presence import RealtimePresence -from ably.realtime.presencemap import PresenceMap -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.presence import PresenceAction, PresenceMessage +from ably.pubsub.realtime.presence import RealtimePresence +from ably.pubsub.realtime.presencemap import PresenceMap +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.presence import PresenceAction, PresenceMessage #: The lowercase action names `RealtimePresence` emits its events under, which come #: from `PresenceAction._action_name` and exist only once `presence.py` is imported. diff --git a/test/uts/helpers/sandbox.py b/test/uts/helpers/sandbox.py index 0fb89fec..a29236c3 100644 --- a/test/uts/helpers/sandbox.py +++ b/test/uts/helpers/sandbox.py @@ -4,7 +4,7 @@ POST the canonical app setup body to the sandbox, take the keys out of the response, and DELETE the app when the tests are done. -Provisioning goes over plain `httpx` rather than through `AblyRest`. It is +Provisioning goes over plain `httpx` rather than through `DefaultPubSubHttpClient`. It is infrastructure, and a client that cannot form a request would otherwise look like a broken fixture rather than a failing test. @@ -80,7 +80,7 @@ def fixture_cipher_params(): key and the IV base64-encoded, which is not what `CipherParams` wants: it takes them as raw bytes and derives the key length from the key. """ - from ably.util.crypto import CipherParams + from ably.pubsub.util.crypto import CipherParams return CipherParams( algorithm=CIPHER_FIXTURE['algorithm'], diff --git a/test/uts/objects/__init__.py b/test/uts/objects/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/uts/objects/helpers/__init__.py b/test/uts/objects/helpers/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/uts/objects/helpers/standard_test_pool.py b/test/uts/objects/helpers/standard_test_pool.py new file mode 100644 index 00000000..7286136a --- /dev/null +++ b/test/uts/objects/helpers/standard_test_pool.py @@ -0,0 +1,514 @@ +"""The fixtures every LiveObjects specification shares. + +Derived from uts/objects/helpers/standard_test_pool.md in ably/specification: the +standard pool of objects, the canonical serials, the ObjectMessage and ProtocolMessage +builders, the synced-channel setup the mock-backed specifications open with, the +negative-assertion quiescence pattern, and REST provisioning for the integration tier. + +Every builder returns the JSON-wire dictionary the specification describes, ready for +`mock_ws.send_to_client(...)`: camelCase keys, numeric operation actions and map +semantics, `json` values as JSON-encoded strings and `bytes` values as base64 strings. +The specifications write actions and semantics by name; here they are the numbers the +wire carries, so pass `LWW` where a specification writes `semantics: "LWW"`. Templates +shared at module level (`STANDARD_POOL_OBJECTS`) must not be mutated; build a variant +instead. A test that drives the msgpack wire converts a dictionary first with +`ObjectMessage.from_dict(d).to_dict('msgpack')`. + +The pure unit specifications construct internal objects rather than wire messages; +`object_message()` and `object_messages()` decode a builder's output into the internal +`ObjectMessage` they take, and `capture_updates()` records the updates an object emits, +since `apply_operation` returns whether it applied rather than the update itself. +""" + +import base64 +import copy +import json +from typing import NamedTuple +from urllib.parse import quote + +from ably.pubsub.liveobjects import publicmessage +from ably.pubsub.liveobjects.objectmessage import ( + WIRE_FORMAT_JSON, + ObjectMessage, + ObjectOperationAction, + ObjectsMapSemantics, +) +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.flags import Flag +from ably.pubsub.util.clock import Clock +from test.uts.helpers.client import poll_until, realtime_client, sandbox_rest_client +from test.uts.helpers.clock import settle +from test.uts.helpers.mock_http import MockHttpClient +from test.uts.helpers.mock_websocket import MockWebSocket + +# --- Canonical constants --------------------------------------------------------------- + +# The siteCode of the harness connection, under which the client applies its own +# operations on ACK +SITE_CODE = 'test-site' + +# The timeserial every standard-pool object and entry is seeded with. Serials compare +# lexicographically (RTLM9e), so every synthetic serial below is defined relative to it. +POOL_SERIAL = 't:0' + +CONNECTION_ID = 'conn-1' +CONNECTION_KEY = 'conn-key-1' + +# RTO10b3's default, which the harness CONNECTED message states explicitly +GC_GRACE_PERIOD = 86400000 + +# The wire values the specifications write by name +HAS_OBJECTS = int(Flag.HAS_OBJECTS) +OBJECT_SUBSCRIBE_FLAG = int(Flag.OBJECT_SUBSCRIBE) +OBJECT_PUBLISH_FLAG = int(Flag.OBJECT_PUBLISH) +LWW = int(ObjectsMapSemantics.LWW) + +ACK = int(ProtocolMessageAction.ACK) +ATTACH = int(ProtocolMessageAction.ATTACH) +ATTACHED = int(ProtocolMessageAction.ATTACHED) +CONNECTED = int(ProtocolMessageAction.CONNECTED) +DETACH = int(ProtocolMessageAction.DETACH) +DETACHED = int(ProtocolMessageAction.DETACHED) +ERROR = int(ProtocolMessageAction.ERROR) +OBJECT = int(ProtocolMessageAction.OBJECT) +OBJECT_SYNC = int(ProtocolMessageAction.OBJECT_SYNC) + +# The modes every standard channel requests +OBJECTS_MODES = (ChannelMode.OBJECT_SUBSCRIBE, ChannelMode.OBJECT_PUBLISH) + +# The X-Ably-Version the objects REST endpoint is called with: protocol v6, whose +# operation payloads (`mapSet`, `counterInc` and the rest) are what the specification's +# provisioning operations are written in +OBJECTS_REST_API_VERSION = '6' + + +# --- Serials ---------------------------------------------------------------------------- + +def ack_serial(msg_serial, index): + """The serial the harness ACKs operation `index` of the publish `msg_serial` with. + + The first publish's first operation is `ack_serial(0, 0) == 't:1:0'`, which sorts after + `POOL_SERIAL`. These serials are recorded in appliedOnAckSerials (RTO9a2a4) and an echo + carrying one is discarded (RTO9a3), so never reuse one as an inbound serial meant to apply. + """ + return f't:{msg_serial + 1}:{index}' + + +def remote_serial(index): + """A serial for a remote MAP_SET or MAP_REMOVE that wins LWW against a pool entry. + + 0-based: `remote_serial(0) == 't:1'`. A bare number such as `'99'` sorts before `'t:0'` + and would be rejected as stale. + """ + return f't:{index + 1}' + + +def below_ack_serial(index): + """A serial that is not an ACK serial, sorting after `POOL_SERIAL` but before `ack_serial(0, 0)`. + + 0-based: `below_ack_serial(9) == 't:0:9'`. + """ + return f't:0:{index}' + + +# --- Values ----------------------------------------------------------------------------- + +def json_value(value): + """An `ObjectData` wire dictionary carrying `value` as JSON (OD2g).""" + return {'json': json.dumps(value)} + + +def bytes_value(value): + """An `ObjectData` wire dictionary carrying `value` as binary (OD2d).""" + return {'bytes': base64.b64encode(value).decode('ascii')} + + +# --- ProtocolMessage builders ----------------------------------------------------------- + +def build_object_sync_message(channel, channel_serial, object_messages): + """An OBJECT_SYNC ProtocolMessage. A `channel_serial` of None leaves it out (RTO5a5).""" + message = {'action': OBJECT_SYNC, 'channel': channel, 'state': list(object_messages)} + if channel_serial is not None: + message['channelSerial'] = channel_serial + return message + + +def build_object_message(channel, object_messages): + """An OBJECT ProtocolMessage.""" + return {'action': OBJECT, 'channel': channel, 'state': list(object_messages)} + + +def build_ack_message(msg_serial, serials): + """An ACK for one ProtocolMessage, carrying a `PublishResult` with `serials` (TR4s).""" + return {'action': ACK, 'msgSerial': msg_serial, 'count': 1, 'res': [{'serials': list(serials)}]} + + +# --- ObjectMessage builders: operations ------------------------------------------------- + +def _operation_message(serial, site_code, operation, serial_timestamp=None): + message = {'serial': serial, 'siteCode': site_code, 'operation': operation} + if serial_timestamp is not None: + message['serialTimestamp'] = serial_timestamp + return message + + +def build_counter_inc(object_id, number, serial, site_code): + return _operation_message(serial, site_code, { + 'action': int(ObjectOperationAction.COUNTER_INC), + 'objectId': object_id, + 'counterInc': {'number': number}, + }) + + +def build_map_set(object_id, key, value, serial, site_code): + """A MAP_SET; `value` is an `ObjectData` wire dictionary, such as `{'string': 'Bob'}`.""" + return _operation_message(serial, site_code, { + 'action': int(ObjectOperationAction.MAP_SET), + 'objectId': object_id, + 'mapSet': {'key': key, 'value': value}, + }) + + +def build_map_remove(object_id, key, serial, site_code, serial_timestamp=None): + return _operation_message(serial, site_code, { + 'action': int(ObjectOperationAction.MAP_REMOVE), + 'objectId': object_id, + 'mapRemove': {'key': key}, + }, serial_timestamp) + + +def build_map_clear(object_id, serial, site_code): + return _operation_message(serial, site_code, { + 'action': int(ObjectOperationAction.MAP_CLEAR), + 'objectId': object_id, + }) + + +def build_object_delete(object_id, serial, site_code, serial_timestamp=None): + return _operation_message(serial, site_code, { + 'action': int(ObjectOperationAction.OBJECT_DELETE), + 'objectId': object_id, + }, serial_timestamp) + + +def build_counter_create(object_id, counter_create, serial, site_code): + """A COUNTER_CREATE; `counter_create` is the wire payload, such as `{'count': 42}`.""" + return _operation_message(serial, site_code, { + 'action': int(ObjectOperationAction.COUNTER_CREATE), + 'objectId': object_id, + 'counterCreate': counter_create, + }) + + +def build_map_create(object_id, map_create, serial, site_code): + """A MAP_CREATE; `map_create` is the wire payload, `{'semantics': LWW, 'entries': {...}}`.""" + return _operation_message(serial, site_code, { + 'action': int(ObjectOperationAction.MAP_CREATE), + 'objectId': object_id, + 'mapCreate': map_create, + }) + + +# --- ObjectMessage builders: state ------------------------------------------------------ + +def build_object_state(object_id, site_timeserials, map=None, counter=None, tombstone=None, create_op=None): + """An OBJECT_SYNC ObjectMessage wrapping an `ObjectState`. + + `map`, `counter`, `tombstone` and `create_op` are the specification's `opts`, as wire + dictionaries. A terse `create_op` such as `{'counterCreate': {...}}` has its mandatory + `objectId` and `action` filled in (OOP2), as the specification's builder does. + """ + state = {'objectId': object_id, 'siteTimeserials': dict(site_timeserials)} + if map is not None: + state['map'] = map + if counter is not None: + state['counter'] = counter + if tombstone is not None: + state['tombstone'] = tombstone + if create_op is not None: + create_op = dict(create_op) + create_op.setdefault('objectId', object_id) + if 'action' not in create_op: + create_op['action'] = int(ObjectOperationAction.COUNTER_CREATE if 'counterCreate' in create_op + else ObjectOperationAction.MAP_CREATE) + state['createOp'] = create_op + return {'object': state} + + +def build_object_message_with_state(object_state): + """An ObjectMessage wrapping an existing `ObjectState` wire dictionary.""" + return {'object': object_state} + + +# The standard tree. Counters carry their value in the create operation, with +# `counter.count` 0, so that applying the state yields 100 and 5 rather than double that. +STANDARD_POOL_OBJECTS = ( + build_object_state('root', {'aaa': POOL_SERIAL}, map={ + 'semantics': LWW, + 'entries': { + 'name': {'data': {'string': 'Alice'}, 'timeserial': POOL_SERIAL}, + 'age': {'data': {'number': 30}, 'timeserial': POOL_SERIAL}, + 'active': {'data': {'boolean': True}, 'timeserial': POOL_SERIAL}, + 'score': {'data': {'objectId': 'counter:score@1000'}, 'timeserial': POOL_SERIAL}, + 'profile': {'data': {'objectId': 'map:profile@1000'}, 'timeserial': POOL_SERIAL}, + 'data': {'data': json_value({'tags': ['a', 'b']}), 'timeserial': POOL_SERIAL}, + 'avatar': {'data': {'bytes': 'AQID'}, 'timeserial': POOL_SERIAL}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:score@1000', {'aaa': POOL_SERIAL}, counter={'count': 0}, + create_op={'counterCreate': {'count': 100}}), + build_object_state('map:profile@1000', {'aaa': POOL_SERIAL}, map={ + 'semantics': LWW, + 'entries': { + 'email': {'data': {'string': 'alice@example.com'}, 'timeserial': POOL_SERIAL}, + 'nested_counter': {'data': {'objectId': 'counter:nested@1000'}, 'timeserial': POOL_SERIAL}, + 'prefs': {'data': {'objectId': 'map:prefs@1000'}, 'timeserial': POOL_SERIAL}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:nested@1000', {'aaa': POOL_SERIAL}, counter={'count': 0}, + create_op={'counterCreate': {'count': 5}}), + build_object_state('map:prefs@1000', {'aaa': POOL_SERIAL}, map={ + 'semantics': LWW, + 'entries': { + 'theme': {'data': {'string': 'dark'}, 'timeserial': POOL_SERIAL}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), +) + + +# --- Internal objects for the pure unit specifications ---------------------------------- + +def object_message(wire, format=WIRE_FORMAT_JSON): + """The internal `ObjectMessage` a builder's wire dictionary decodes to.""" + return ObjectMessage.from_dict(wire, format) + + +def object_messages(protocol_message, format=WIRE_FORMAT_JSON): + """The internal ObjectMessages an OBJECT or OBJECT_SYNC ProtocolMessage's `state` decodes to.""" + return ObjectMessage.from_protocol_message(protocol_message, format) + + +def capture_updates(live_object): + """Records every update `live_object` emits, the no-ops included, and returns the list. + + `apply_operation` returns whether the operation applied (RTLC7g, RTLM15g) and emits the + resulting update through `notify_updated`, so a specification's + `update = obj.applyOperation(...)` is `updates[-1]` after the call. The object's own + `notify_updated` still runs. + """ + updates = [] + original = live_object.notify_updated + + def notify_updated(update): + updates.append(update) + original(update) + + live_object.notify_updated = notify_updated + return updates + + +def build_public_object_message(message, channel_name): + """The public `ObjectMessage` PAOM3 derives from `message` received on `channel_name`. + + `message` is an internal `ObjectMessage` or a builder's wire dictionary. This is the + expected value a subscription test compares against, built here independently of the + library's own derivation. + """ + if isinstance(message, dict): + message = object_message(message) + operation = message.operation + public_operation = publicmessage.ObjectOperation( + action=operation.action, + object_id=operation.object_id, + map_create=operation.resolved_map_create, + map_set=operation.map_set, + map_remove=operation.map_remove, + counter_create=operation.resolved_counter_create, + counter_inc=operation.counter_inc, + object_delete=operation.object_delete, + map_clear=operation.map_clear, + ) + return publicmessage.ObjectMessage( + channel=channel_name, + operation=public_operation, + id=message.id, + client_id=message.client_id, + connection_id=message.connection_id, + timestamp=message.timestamp, + serial=message.serial, + serial_timestamp=message.serial_timestamp, + site_code=message.site_code, + extras=message.extras, + ) + + +# --- The mock-backed client ------------------------------------------------------------- + +def objects_connected_message(site_code=SITE_CODE, objects_gc_grace_period=GC_GRACE_PERIOD, + connection_id=CONNECTION_ID, connection_key=CONNECTION_KEY): + """The harness CONNECTED message, carrying `siteCode` and `objectsGCGracePeriod`. + + Passing None for either leaves it out. `connectionId` travels on the ProtocolMessage + itself, which is where the library reads it, though the specification writes it among + the `connectionDetails`. `maxIdleInterval` is 0 so that the transport schedules no + idle timer, which a `FakeClock` test would otherwise fire. + """ + details = {'connectionKey': connection_key, 'connectionStateTtl': 120000, 'maxIdleInterval': 0} + if site_code is not None: + details['siteCode'] = site_code + if objects_gc_grace_period is not None: + details['objectsGCGracePeriod'] = objects_gc_grace_period + return {'action': CONNECTED, 'connectionId': connection_id, 'connectionDetails': details} + + +def objects_attached_message(channel, channel_serial='sync1:', flags=HAS_OBJECTS): + """An ATTACHED for `channel`. + + The specifications write the modes a server grants as `modes: [...]`; on the wire they + are bits of `flags`, so pass `HAS_OBJECTS | OBJECT_SUBSCRIBE_FLAG` for + `flags: HAS_OBJECTS, modes: ["OBJECT_SUBSCRIBE"]`. + """ + return {'action': ATTACHED, 'channel': channel, 'channelSerial': channel_serial, 'flags': flags} + + +def standard_mock_websocket(auto_ack=True, on_object=None, connected=None, attached_channel_serial='sync1:', + attached_flags=HAS_OBJECTS, sync_objects=STANDARD_POOL_OBJECTS, + sync_channel_serial='sync1:'): + """The specification's synced-channel `MockWebSocket`. + + It accepts every connection with `connected` (by default `objects_connected_message()`); + answers an ATTACH with an ATTACHED and then an OBJECT_SYNC of `sync_objects`, or no + OBJECT_SYNC if `sync_objects` is None; answers a DETACH with a DETACHED; and passes each + OBJECT the client sends to `on_object`, then, with `auto_ack`, ACKs it with + `ack_serial(msgSerial, i)` for each of its operations. + """ + connected_message = connected if connected is not None else objects_connected_message() + mock_ws = MockWebSocket(on_connection_attempt=lambda conn: conn.respond_with_success(connected_message)) + + def on_message_from_client(message): + action = message.get('action') + if action == ATTACH: + channel = message.get('channel') + mock_ws.send_to_client(objects_attached_message(channel, attached_channel_serial, attached_flags)) + if sync_objects is not None: + mock_ws.send_to_client(build_object_sync_message(channel, sync_channel_serial, sync_objects)) + elif action == OBJECT: + if on_object is not None: + on_object(message) + if auto_ack: + msg_serial = message['msgSerial'] + serials = [ack_serial(msg_serial, i) for i in range(len(message.get('state') or []))] + mock_ws.send_to_client(build_ack_message(msg_serial, serials)) + elif action == DETACH: + mock_ws.send_to_client({'action': DETACHED, 'channel': message.get('channel')}) + + mock_ws.on_message_from_client = on_message_from_client + return mock_ws + + +def time_mock_http(clock=None): + """A `MockHttpClient` answering `GET /time` with the time on `clock`. + + Creating an object reads the server time (RTLCV4e, RTLMV4h, RTO16), and a mock-backed + client must not reach the network for it. Any other request is answered 404. + """ + time_source = clock if clock is not None else Clock() + + def on_request(request): + if request.method == 'GET' and request.path == '/time': + request.respond_with(200, [time_source.now_ms()]) + else: + request.respond_with(404, {'error': {'code': 40400, 'statusCode': 404, 'message': 'Not found'}}) + + return MockHttpClient(on_connection_attempt=lambda conn: conn.respond_with_success(), on_request=on_request) + + +def objects_client(mock_ws, clock=None, mock_http=None, **kwargs): + """A realtime client on `mock_ws`, as the specifications' `Realtime(options: {key: ...})`. + + It connects on its own, speaks JSON so that the builders' dictionaries go over the mock + as they are, and has `GET /time` answered by `time_mock_http(clock)` unless `mock_http` + is given. `clock` is a `FakeClock` for a specification's `enable_fake_timers()`. + """ + kwargs.setdefault('auto_connect', True) + kwargs.setdefault('use_binary_protocol', False) + if mock_http is None: + mock_http = time_mock_http(clock) + return realtime_client(mock_ws, mock_http=mock_http, clock=clock, **kwargs) + + +def objects_channel_options(*modes): + """`ChannelOptions` requesting `modes`, by default OBJECT_SUBSCRIBE and OBJECT_PUBLISH.""" + return ChannelOptions(modes=list(modes or OBJECTS_MODES)) + + +class SyncedChannel(NamedTuple): + """What `setup_synced_channel` returns. It unpacks as `client, channel, root, mock_ws`.""" + + client: object + channel: object + root: object + mock_ws: MockWebSocket + + +async def setup_synced_channel(channel_name='test', mock_ws=None, clock=None, modes=OBJECTS_MODES, + **client_options): + """The specification's `setup_synced_channel(channel_name)`. + + Connects a client to `mock_ws` (by default `standard_mock_websocket()`), gets + `channel_name` with `modes`, and awaits `channel.object.get()`, which attaches the + channel and waits for the standard pool to sync. `clock` and `client_options` go to + `objects_client`. + """ + if mock_ws is None: + mock_ws = standard_mock_websocket() + client = objects_client(mock_ws, clock=clock, **client_options) + channel = client.channels.get(channel_name, objects_channel_options(*modes)) + root = await channel.object.get() + return SyncedChannel(client, channel, root, mock_ws) + + +async def setup_synced_channel_no_ack(channel_name='test', **kwargs): + """The specification's `setup_synced_channel_no_ack`: OBJECT messages are recorded but not ACKed.""" + return await setup_synced_channel(channel_name, mock_ws=standard_mock_websocket(auto_ack=False), **kwargs) + + +# --- Negative-assertion quiescence ------------------------------------------------------ + +async def assert_unchanged_after_quiescence(count_under_test, control_delivered, timeout=5.0, + description='the control to be delivered'): + """The specification's `assert_unchanged_after_quiescence(count_under_test, control)`. + + Reads `count_under_test()`, waits until `control_delivered()` holds, which a control + listener (or a follow-up message) on the same dispatch makes true, settles, and asserts + the count has not moved. Send the message under test before calling this, and the + control's stimulus after it. + """ + before = count_under_test() + await poll_until(control_delivered, timeout=timeout, description=description) + await settle() + after = count_under_test() + assert after == before, f'expected the count to stay at {before} once {description}, but it was {after}' + + +# --- REST provisioning for the integration tier ---------------------------------------- + +async def provision_objects_via_rest(api_key, channel_name, operations, client=None): + """POSTs `operations` to the channel's objects REST endpoint, before any realtime client connects. + + `operations` is one operation dictionary or a list of them, in the REST API's format: + `{'mapSet': {'key': ..., 'value': {...}}, 'objectId': 'root'}` and so on. Returns the + `objectIds` of every result, flattened in request order. The REST client goes straight + to the sandbox and is closed when the test ends; `client` replaces it. + """ + if client is None: + client = sandbox_rest_client(api_key) + path = f"/channels/{quote(channel_name, safe='')}/object" + response = await client.request('POST', path, OBJECTS_REST_API_VERSION, body=copy.deepcopy(operations)) + if not response.success: + raise AssertionError( + f'Provisioning objects on {channel_name!r} failed: {response.status_code} ' + f'{response.error_code} {response.error_message}') + return [object_id for result in response.items for object_id in (result.get('objectIds') or [])] diff --git a/test/uts/objects/helpers/standard_test_pool_test.py b/test/uts/objects/helpers/standard_test_pool_test.py new file mode 100644 index 00000000..0b2bfbd5 --- /dev/null +++ b/test/uts/objects/helpers/standard_test_pool_test.py @@ -0,0 +1,292 @@ +"""Tests for the `standard_test_pool` helper and the object wire types it builds on. + +These are not derived from a specification. They pin the builders to the wire shapes +`standard_test_pool.md` describes, the canonical serials to the ordering the +specifications rely on, the wire types to a lossless round trip on both protocols, and +the mock to the synced-channel conversation, which it can hold before any LiveObjects +behaviour exists. +""" + +import json + +import msgpack + +from ably.pubsub.liveobjects import publicmessage +from ably.pubsub.liveobjects.objectmessage import ( + WIRE_FORMAT_MSGPACK, + CounterCreate, + MapCreate, + MapCreateWithObjectId, + ObjectData, + ObjectMessage, + ObjectOperation, + ObjectOperationAction, + ObjectsMapSemantics, +) +from ably.pubsub.liveobjects.realtimeobject import RealtimeObject +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.channelmode import ChannelMode, decode_channel_mode +from ably.pubsub.types.channelstate import ChannelState +from test.uts.helpers.client import await_channel_state, await_connection_state, poll_until, rest_client +from test.uts.helpers.clock import settle +from test.uts.helpers.mock_http import MockHttpClient +from test.uts.helpers.mock_websocket import MockEventType +from test.uts.objects.helpers.standard_test_pool import ( + GC_GRACE_PERIOD, + HAS_OBJECTS, + LWW, + OBJECT_SUBSCRIBE_FLAG, + OBJECT_SYNC, + POOL_SERIAL, + SITE_CODE, + STANDARD_POOL_OBJECTS, + ack_serial, + assert_unchanged_after_quiescence, + below_ack_serial, + build_ack_message, + build_counter_inc, + build_map_remove, + build_map_set, + build_object_delete, + build_object_message, + build_object_state, + build_object_sync_message, + build_public_object_message, + bytes_value, + capture_updates, + json_value, + object_message, + object_messages, + objects_channel_options, + objects_client, + provision_objects_via_rest, + remote_serial, + standard_mock_websocket, +) + + +def msgpack_round_trip(message): + """`message` encoded for the msgpack wire, sent through msgpack and decoded again.""" + raw = msgpack.packb(message.to_dict(WIRE_FORMAT_MSGPACK), use_bin_type=True) + return ObjectMessage.from_dict(msgpack.unpackb(raw, raw=False), WIRE_FORMAT_MSGPACK) + + +def test_serials_sort_as_the_specification_requires(): + assert ack_serial(0, 0) == 't:1:0' + assert remote_serial(0) == 't:1' + assert below_ack_serial(9) == 't:0:9' + # A remote write and the first ACK both beat the pool, and the probe sits between them + assert POOL_SERIAL < below_ack_serial(9) < ack_serial(0, 0) + assert remote_serial(0) > POOL_SERIAL + + +def test_operation_builders_produce_the_json_wire_shape(): + assert build_counter_inc('counter:a@1', 5, '01', 'site1') == { + 'serial': '01', 'siteCode': 'site1', + 'operation': {'action': 4, 'objectId': 'counter:a@1', 'counterInc': {'number': 5}}, + } + assert build_map_set('root', 'k', {'string': 'v'}, '02', 'site1')['operation'] == { + 'action': 1, 'objectId': 'root', 'mapSet': {'key': 'k', 'value': {'string': 'v'}}, + } + assert 'serialTimestamp' not in build_map_remove('root', 'k', '03', 'site1') + assert build_object_delete('counter:a@1', '04', 'site1', 1700000000000)['serialTimestamp'] == 1700000000000 + + +def test_protocol_message_builders(): + sync = build_object_sync_message('test', 'sync1:', STANDARD_POOL_OBJECTS) + assert sync['action'] == OBJECT_SYNC + assert sync['channelSerial'] == 'sync1:' + assert len(sync['state']) == 5 + # RTO5a5: a sync with no channelSerial carries no key at all + assert 'channelSerial' not in build_object_sync_message('test', None, []) + assert build_object_message('test', [])['action'] == 19 + assert build_ack_message(3, ['a', None]) == {'action': 1, 'msgSerial': 3, 'count': 1, + 'res': [{'serials': ['a', None]}]} + + +def test_object_state_builder_fills_in_the_create_op(): + counter = build_object_state('counter:a@1', {'s': '1'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 7}}) + assert counter['object']['createOp'] == {'counterCreate': {'count': 7}, 'objectId': 'counter:a@1', + 'action': int(ObjectOperationAction.COUNTER_CREATE)} + a_map = build_object_state('map:a@1', {}, map={'semantics': LWW, 'entries': {}}, + create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}, tombstone=True) + assert a_map['object']['createOp']['action'] == int(ObjectOperationAction.MAP_CREATE) + assert a_map['object']['tombstone'] is True + assert 'tombstone' not in counter['object'] + + +def test_standard_pool_decodes_to_the_specified_tree(): + root, score, profile, nested, prefs = (object_message(message).object for message in STANDARD_POOL_OBJECTS) + + assert root.object_id == 'root' + assert root.site_timeserials == {'aaa': POOL_SERIAL} + assert root.map.semantics is ObjectsMapSemantics.LWW + entries = root.map.entries + assert entries['name'].data == ObjectData(string='Alice') + assert entries['age'].data == ObjectData(number=30.0) + assert entries['active'].data == ObjectData(boolean=True) + assert entries['score'].data == ObjectData(object_id='counter:score@1000') + assert entries['data'].data.json == {'tags': ['a', 'b']} + assert entries['avatar'].data.bytes == bytes([1, 2, 3]) + assert all(entry.timeserial == POOL_SERIAL and not entry.tombstone for entry in entries.values()) + assert root.create_op.action is ObjectOperationAction.MAP_CREATE + assert root.create_op.object_id == 'root' + + assert score.counter.count == 0 + assert score.create_op.counter_create == CounterCreate(count=100.0) + assert nested.create_op.counter_create.count == 5 + assert set(profile.map.entries) == {'email', 'nested_counter', 'prefs'} + assert prefs.map.entries['theme'].data.string == 'dark' + + +def test_values_carry_json_and_binary_as_the_json_wire_does(): + assert json_value({'a': [1]}) == {'json': '{"a": [1]}'} + assert bytes_value(bytes([1, 2, 3])) == {'bytes': 'AQID'} + data = ObjectData.from_dict({'json': '{"a": [1]}'}) + assert data.json == {'a': [1]} + assert data.to_dict() == {'json': '{"a":[1]}'} + + +def test_wire_types_round_trip_through_json(): + for wire in [*STANDARD_POOL_OBJECTS, build_counter_inc('counter:a@1', 5, '01', 'site1'), + build_map_set('root', 'k', bytes_value(b'\x00\xff'), '02', 'site1'), + build_map_remove('root', 'k', '03', 'site1', 1700000000000)]: + decoded = object_message(wire) + assert object_message(json.loads(json.dumps(decoded.to_dict()))) == decoded + + +def test_wire_types_round_trip_through_msgpack(): + for wire in [*STANDARD_POOL_OBJECTS, build_map_set('root', 'k', bytes_value(b'\x00\xff'), '02', 'site1')]: + decoded = object_message(wire) + assert msgpack_round_trip(decoded) == decoded + # Binary is raw on the msgpack wire and base64 on the JSON one (OD2d) + avatar = object_message(STANDARD_POOL_OBJECTS[0]).object.map.entries['avatar'].data + assert avatar.to_dict(WIRE_FORMAT_MSGPACK) == {'bytes': bytes([1, 2, 3])} + assert avatar.to_dict() == {'bytes': 'AQID'} + + +def test_local_only_fields_are_never_encoded(): + derived = MapCreate(semantics=ObjectsMapSemantics.LWW, entries={}) + operation = ObjectOperation( + action=ObjectOperationAction.MAP_CREATE, object_id='map:a@1', + map_create_with_object_id=MapCreateWithObjectId(initial_value='{}', nonce='n' * 16, derived_from=derived)) + assert operation.to_dict() == { + 'action': 0, 'objectId': 'map:a@1', 'mapCreateWithObjectId': {'initialValue': '{}', 'nonce': 'n' * 16}, + } + assert operation.resolved_map_create is derived + entry = object_message(STANDARD_POOL_OBJECTS[0]).object.map.entries['name'] + entry.tombstoned_at = 1700000000000 + assert 'tombstonedAt' not in json.dumps(entry.to_dict()) + + +def test_unknown_action_decodes_without_failing(): + message = object_message({'serial': '01', 'siteCode': 's', 'operation': {'action': 99, 'objectId': 'x'}}) + assert message.operation.action is ObjectOperationAction.UNKNOWN + + +def test_object_messages_take_identity_from_the_protocol_message(): + protocol_message = build_object_message('test', [build_counter_inc('counter:a@1', 1, '01', 's'), + build_counter_inc('counter:a@1', 2, '02', 's')]) + protocol_message.update(id='pm-id', connectionId='conn-x', timestamp=1234) + first, second = object_messages(protocol_message) + assert (first.id, second.id) == ('pm-id:0', 'pm-id:1') + assert first.connection_id == 'conn-x' + assert second.timestamp == 1234 + + +def test_public_object_message_follows_paom3(): + wire = build_counter_inc('counter:score@1000', 42, 'serial-1', 'site-a') + wire['clientId'] = 'client-1' + public = build_public_object_message(wire, 'test') + assert isinstance(public, publicmessage.ObjectMessage) + assert public.channel == 'test' + assert public.serial == 'serial-1' + assert public.site_code == 'site-a' + assert public.client_id == 'client-1' + assert public.operation.action is ObjectOperationAction.COUNTER_INC + assert public.operation.counter_inc.number == 42 + assert public.operation.map_create is None + + +def test_capture_updates_records_and_delegates(): + delivered = [] + + class Recorder: + def notify_updated(self, update): + delivered.append(update) + + recorder = Recorder() + updates = capture_updates(recorder) + recorder.notify_updated('first') + recorder.notify_updated('second') + assert updates == ['first', 'second'] + assert delivered == ['first', 'second'] + + +async def test_quiescence_passes_when_only_the_control_moves(): + under_test = [] + control = ['delivered'] + await assert_unchanged_after_quiescence(lambda: len(under_test), lambda: len(control) >= 1) + + +async def test_standard_mock_holds_the_synced_channel_conversation(): + mock_ws = standard_mock_websocket() + client = objects_client(mock_ws) + await await_connection_state(client, ConnectionState.CONNECTED) + + details = client.connection.connection_details + assert details.site_code == SITE_CODE + assert details.objects_gc_grace_period == GC_GRACE_PERIOD + + channel = client.channels.get('test', objects_channel_options()) + await channel.attach() + await poll_until(lambda: any(event.data.get('action') == OBJECT_SYNC + for event in mock_ws.events_of_type(MockEventType.MESSAGE_TO_CLIENT)), + description='the OBJECT_SYNC to follow the ATTACHED') + attach = next(m for m in mock_ws.messages_from_client if m.get('channel') == 'test') + assert decode_channel_mode(attach['flags']) == [ChannelMode.OBJECT_SUBSCRIBE, ChannelMode.OBJECT_PUBLISH] + + await channel.detach() + await await_channel_state(channel, ChannelState.DETACHED) + + +async def test_granted_modes_decode_from_the_attached_flags(): + mock_ws = standard_mock_websocket(attached_flags=HAS_OBJECTS | OBJECT_SUBSCRIBE_FLAG, sync_objects=None) + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options()) + await channel.attach() + assert channel.modes == [ChannelMode.OBJECT_SUBSCRIBE] + await settle() + assert not any(event.data.get('action') == OBJECT_SYNC + for event in mock_ws.events_of_type(MockEventType.MESSAGE_TO_CLIENT)) + + +async def test_each_channel_has_one_realtime_object(): + client = objects_client(standard_mock_websocket(), auto_connect=False) + channel = client.channels.get('test') + assert isinstance(channel.object, RealtimeObject) + assert channel.object is channel.object + assert channel.object._channel is channel + assert 'root' in channel.object._objects_pool + + +async def test_provisioning_posts_operations_and_flattens_object_ids(): + captured = [] + + def on_request(request): + captured.append(request) + request.respond_with(201, [{'objectIds': ['root', 'counter:a@1']}, {'objectIds': ['map:b@2']}]) + + mock_http = MockHttpClient(on_connection_attempt=lambda conn: conn.respond_with_success(), + on_request=on_request) + client = rest_client(mock_http, use_binary_protocol=False) + operations = [{'mapSet': {'key': 'k', 'value': {'string': 'v'}}, 'objectId': 'root'}, + {'counterCreate': {'count': 1}}] + + object_ids = await provision_objects_via_rest(None, 'a channel:1', operations, client=client) + + assert object_ids == ['root', 'counter:a@1', 'map:b@2'] + assert captured[0].method == 'POST' + assert captured[0].url.raw_path == '/channels/a%20channel%3A1/object' + assert json.loads(captured[0].body) == operations diff --git a/test/uts/objects/integration/__init__.py b/test/uts/objects/integration/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/uts/objects/integration/conftest.py b/test/uts/objects/integration/conftest.py new file mode 100644 index 00000000..62ee721c --- /dev/null +++ b/test/uts/objects/integration/conftest.py @@ -0,0 +1,61 @@ +"""Fixtures the LiveObjects integration specifications share. + +The sandbox app is provisioned once and read by every test that asks for it, as +it is for the realtime integration tier: each specification's `BEFORE ALL TESTS` +provisions an app, and provisioning one per file would make the tier slower and +invite the sandbox's rate limiting. The fixtures in +`test/uts/realtime/integration/conftest.py` are not visible from this package, +so the same shape is defined here; the app is a separate one from the realtime +tier's, so that a channel one tier writes objects to is never one the other +reads. +""" + +import os + +import pytest +import pytest_asyncio + +from test.uts.helpers.sandbox import delete_app, provision_app + +# `integration-testing.md` puts a suite of this size at 120 seconds. A LiveObjects +# test opens one or two connections, attaches, waits for the objects sync and then +# waits again for a mutation to reach a second client, each bounded at 10 to 30 +# seconds. The repository default in `pyproject.toml` is 30 seconds, which suits a +# test served from a mock; the marker applies to this package alone. +SUITE_TIMEOUT = 120 + +__package_dir = os.path.dirname(os.path.abspath(__file__)) + + +def pytest_collection_modifyitems(items): + for item in items: + if os.path.abspath(str(item.fspath)).startswith(__package_dir + os.sep): + item.add_marker(pytest.mark.timeout(SUITE_TIMEOUT)) + + +@pytest_asyncio.fixture(scope='session') +async def realtime_sandbox(): + """The provisioned sandbox app, as a specification's `app_config`. + + This is the specifications' `BEFORE ALL TESTS` / `AFTER ALL TESTS` pair. + `realtime_sandbox.key_str` is the full-access key they call `api_key`. + """ + app = await provision_app() + yield app + await delete_app(app) + + +@pytest.fixture(params=[False, True], ids=['json', 'msgpack']) +def use_binary_protocol(request): + """Runs the test once per protocol, which is the specifications' `PROTOCOL`. + + The three specifications directly in this package carry a + `## Protocol Variants` section, so their tests run against both json and + msgpack and pass this straight to every client they build: + + client = sandbox_realtime_client(api_key, use_binary_protocol=use_binary_protocol) + + The proxy specification has no such section and runs json only, which the + proxy requires in any case. + """ + return request.param diff --git a/test/uts/objects/integration/objects_gc_test.py b/test/uts/objects/integration/objects_gc_test.py new file mode 100644 index 00000000..f6182c1b --- /dev/null +++ b/test/uts/objects/integration/objects_gc_test.py @@ -0,0 +1,109 @@ +"""Derived from uts/objects/integration/objects_gc_test.md in ably/specification. + +Spec points: RTO10, RTLM19, RTLM5d2h, RTLM7 + +Tombstone semantics end to end against the sandbox: removing a map entry tombstones it +(RTLM7), a tombstoned entry reads back as absent (RTLM5d2h), and the key can be set again, +with a fresh object id where the new value is an object (RTO10, RTLM19). Only the public +API is used. + +The GC sweep itself is not exercised here, as the specification's Scope section says: its +five-minute cadence and the server's grace period are not observable within an +integration test's budget, so the sweep is covered at the unit tier against a `FakeClock`. +Every wait here is on real time. + +The specification carries a `## Protocol Variants` section, so every test runs once per +protocol and passes `use_binary_protocol` to the client it builds. + +The specification's `value() == null` is `None` in Python. A counter's `value()` is reached +through `as_live_counter()` and a primitive's through `as_primitive()`, since ably-python's +`PathObject` is partitioned by type (RTTS); either view answers `None` once the path no +longer resolves (RTTS5d1), and each test has already seen a value at that path, so the +`None` is the removal rather than a value that never arrived. Clients are closed by the +suite's fixture. +""" + +import asyncio + +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server.liveobjects import LiveCounter +from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until +from test.uts.helpers.sandbox import random_id +from test.uts.objects.helpers.standard_test_pool import objects_channel_options + +# The specification's `AWAIT_STATE ... WITH timeout` and `AWAIT channel.object.get() WITH +# timeout`, in seconds. +CONNECT_TIMEOUT = 15.0 +SYNC_TIMEOUT = 15.0 + + +async def synced_root(api_key, channel_name, use_binary_protocol): + """The setup both tests share: a CONNECTED client and the synced root of `channel_name`.""" + client = sandbox_realtime_client(api_key, auto_connect=False, use_binary_protocol=use_binary_protocol) + client.connect() + await await_connection_state(client, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + + channel = client.channels.get(channel_name, objects_channel_options()) + try: + return await asyncio.wait_for(channel.object.get(), SYNC_TIMEOUT) + except asyncio.TimeoutError: + raise AssertionError( + f'Timed out after {SYNC_TIMEOUT}s waiting for channel.object.get() on {channel_name!r}') from None + + +# UTS: objects/integration/RTO10/tombstoned-object-gc-recreate-0 +async def test_rto10_tombstoned_object_gc_recreate(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-gc-object-' + random_id() + root = await synced_root(realtime_sandbox.key_str, channel_name, use_binary_protocol) + + # Create a counter + await root.set('counter', LiveCounter.create(42)) + await wall_clock_poll_until( + lambda: root.get('counter').as_live_counter().value() == 42, + description='counter to read 42') + + counter_id = root.get('counter').instance().id + + # Remove it (tombstones the entry and the object, RTLM7) + await root.remove('counter') + + # RTLM5d2h: tombstoned entries read back as None + await wall_clock_poll_until( + lambda: root.get('counter').as_live_counter().value() is None, + description='the removed counter to read None') + + # Create a new counter at the same key + await root.set('counter', LiveCounter.create(99)) + await wall_clock_poll_until( + lambda: root.get('counter').as_live_counter().value() == 99, + description='the new counter to read 99') + + assert root.get('counter').as_live_counter().value() == 99 + assert root.get('counter').instance().id != counter_id + + +# UTS: objects/integration/RTLM19/tombstoned-entry-gc-reset-0 +async def test_rtlm19_tombstoned_entry_gc_reset(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-gc-entry-' + random_id() + root = await synced_root(realtime_sandbox.key_str, channel_name, use_binary_protocol) + + # Set then remove a key + await root.set('ephemeral', 'temporary') + await wall_clock_poll_until( + lambda: root.get('ephemeral').as_primitive().value() == 'temporary', + description="ephemeral to read 'temporary'") + + await root.remove('ephemeral') + + # RTLM5d2h: tombstoned entries read back as None + await wall_clock_poll_until( + lambda: root.get('ephemeral').as_primitive().value() is None, + description='the removed entry to read None') + + # Set the same key again + await root.set('ephemeral', 'revived') + await wall_clock_poll_until( + lambda: root.get('ephemeral').as_primitive().value() == 'revived', + description="ephemeral to read 'revived'") + + assert root.get('ephemeral').as_primitive().value() == 'revived' diff --git a/test/uts/objects/integration/objects_lifecycle_test.py b/test/uts/objects/integration/objects_lifecycle_test.py new file mode 100644 index 00000000..98b87965 --- /dev/null +++ b/test/uts/objects/integration/objects_lifecycle_test.py @@ -0,0 +1,167 @@ +"""Derived from uts/objects/integration/objects_lifecycle_test.md in ably/specification. + +Spec points: RTO23, RTPO15, RTPO17 + +End to end against the sandbox: connect, sync, create and mutate objects through the +root `PathObject`, and read the result from a second client. Only the public API is used. + +The specification carries a `## Protocol Variants` section, so every test runs once per +protocol and passes `use_binary_protocol` to each client it builds. Provisioning over REST +is a JSON request whichever protocol the realtime client speaks. + +The specification writes its reads and writes against an untyped `PathObject`. +ably-python's `PathObject` is partitioned by type (RTTS), so a counter's `value()` and +`increment()` are reached through `as_live_counter()` and a primitive's `value()` through +`as_primitive()`; `root` is already a `LiveMapPathObject`, so `set` and `size` are called on +it directly. That is a difference of spelling, not of behaviour. + +`AWAIT_STATE` for CONNECTED waits ten seconds, the figure the realtime integration tier +uses for a connect over the network. The specification leaves `channel.object.get()` +unbounded; it is bounded here at the fifteen seconds `objects_gc_test.md` gives the same +call, so that a sync that never completes fails with a message naming it rather than at the +package timeout. Clients are closed by the suite's fixture. +""" + +import asyncio + +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server.liveobjects import LiveCounter, LiveMap, PathObject +from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until +from test.uts.helpers.sandbox import random_id +from test.uts.objects.helpers.standard_test_pool import objects_channel_options, provision_objects_via_rest + +CONNECT_TIMEOUT = 10.0 +SYNC_TIMEOUT = 15.0 + + +async def connected_client(api_key, use_binary_protocol): + """The specification's `Realtime(options: {...})`, connected and CONNECTED.""" + client = sandbox_realtime_client(api_key, auto_connect=False, use_binary_protocol=use_binary_protocol) + client.connect() + await await_connection_state(client, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + return client + + +async def synced_root(channel): + """The specification's `AWAIT channel.object.get()`, bounded at `SYNC_TIMEOUT`.""" + try: + return await asyncio.wait_for(channel.object.get(), SYNC_TIMEOUT) + except asyncio.TimeoutError: + raise AssertionError( + f'Timed out after {SYNC_TIMEOUT}s waiting for channel.object.get() on {channel.name!r}') from None + + +async def synced_pair(api_key, channel_name, use_binary_protocol): + """The two clients, channels and roots every propagation test opens with.""" + client_a = await connected_client(api_key, use_binary_protocol) + client_b = await connected_client(api_key, use_binary_protocol) + + channel_a = client_a.channels.get(channel_name, objects_channel_options()) + channel_b = client_b.channels.get(channel_name, objects_channel_options()) + + root_a = await synced_root(channel_a) + root_b = await synced_root(channel_b) + return root_a, root_b + + +# UTS: objects/integration/RTO23-RTPO15/set-primitive-propagates-0 +async def test_rto23_rtpo15_set_primitive_propagates(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-lifecycle-' + random_id() + root_a, root_b = await synced_pair(realtime_sandbox.key_str, channel_name, use_binary_protocol) + + # Client A sets a value + await root_a.set('greeting', 'hello') + + # Client B subscribes and waits for the update + events_b = [] + root_b.subscribe(events_b.append) + await wall_clock_poll_until( + lambda: root_b.get('greeting').as_primitive().value() == 'hello', + description="client B to read greeting == 'hello'") + + assert root_b.get('greeting').as_primitive().value() == 'hello' + + +# UTS: objects/integration/RTPO15/set-counter-value-type-0 +async def test_rtpo15_set_counter_value_type(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-counter-create-' + random_id() + root_a, root_b = await synced_pair(realtime_sandbox.key_str, channel_name, use_binary_protocol) + + await root_a.set('my_counter', LiveCounter.create(42)) + await wall_clock_poll_until( + lambda: root_b.get('my_counter').as_live_counter().value() == 42, + description='client B to read my_counter == 42') + + assert root_b.get('my_counter').as_live_counter().value() == 42 + assert root_b.get('my_counter').instance() is not None + + +# UTS: objects/integration/RTPO17/increment-propagates-0 +async def test_rtpo17_increment_propagates(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-increment-' + random_id() + root_a, root_b = await synced_pair(realtime_sandbox.key_str, channel_name, use_binary_protocol) + + # Create a counter first + await root_a.set('hits', LiveCounter.create(0)) + await wall_clock_poll_until( + lambda: root_b.get('hits').as_live_counter().value() == 0, + description='client B to read hits == 0') + + # Increment it + await root_a.get('hits').as_live_counter().increment(10) + await wall_clock_poll_until( + lambda: root_b.get('hits').as_live_counter().value() == 10, + description='client B to read hits == 10') + + assert root_a.get('hits').as_live_counter().value() == 10 + assert root_b.get('hits').as_live_counter().value() == 10 + + +# UTS: objects/integration/RTPO15/set-map-value-type-0 +async def test_rtpo15_set_map_value_type(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-map-create-' + random_id() + root_a, root_b = await synced_pair(realtime_sandbox.key_str, channel_name, use_binary_protocol) + + await root_a.set('settings', LiveMap.create({ + 'theme': 'dark', + 'fontSize': 14, + })) + await wall_clock_poll_until( + lambda: root_b.get('settings').get('theme').as_primitive().value() == 'dark', + description="client B to read settings.theme == 'dark'") + + assert root_b.get('settings').get('theme').as_primitive().value() == 'dark' + assert root_b.get('settings').get('fontSize').as_primitive().value() == 14 + + +# UTS: objects/integration/RTO23/get-returns-path-object-0 +async def test_rto23_get_returns_path_object(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-get-root-' + random_id() + + client = await connected_client(realtime_sandbox.key_str, use_binary_protocol) + channel = client.channels.get(channel_name, objects_channel_options()) + + root = await synced_root(channel) + + assert isinstance(root, PathObject) + assert root.path() == '' + assert root.size() == 0 + + +# UTS: objects/integration/RTPO15/rest-provisioned-data-sync-0 +async def test_rtpo15_rest_provisioned_data_sync(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-rest-provision-' + random_id() + + # Provision data via REST before any realtime client connects + await provision_objects_via_rest(realtime_sandbox.key_str, channel_name, [ + { + 'mapSet': {'key': 'provisioned', 'value': {'string': 'from_rest'}}, + 'objectId': 'root', + }, + ]) + + client = await connected_client(realtime_sandbox.key_str, use_binary_protocol) + channel = client.channels.get(channel_name, objects_channel_options()) + root = await synced_root(channel) + + assert root.get('provisioned').as_primitive().value() == 'from_rest' diff --git a/test/uts/objects/integration/objects_sync_test.py b/test/uts/objects/integration/objects_sync_test.py new file mode 100644 index 00000000..23ff6872 --- /dev/null +++ b/test/uts/objects/integration/objects_sync_test.py @@ -0,0 +1,124 @@ +"""Derived from uts/objects/integration/objects_sync_test.md in ably/specification. + +Spec points: RTO4, RTO5, RTO17 + +The objects sync sequence against the sandbox: attaching with object modes gets an +ATTACHED carrying HAS_OBJECTS and an OBJECT_SYNC, after which `channel.object.get()` +resolves; a second client syncs what a first one wrote; and a detach and re-attach syncs +the pool afresh. Only the public API is used. + +The specification carries a `## Protocol Variants` section, so every test runs once per +protocol and passes `use_binary_protocol` to each client it builds. + +A primitive's `value()` is reached through `as_primitive()`, since ably-python's +`PathObject` is partitioned by type (RTTS); `root` is a `LiveMapPathObject`, so `set` and +`size` are called on it directly. + +`AWAIT_STATE` for CONNECTED waits ten seconds, the figure the realtime integration tier +uses for a connect over the network, and `channel.object.get()` is bounded at the fifteen +seconds `objects_gc_test.md` gives the same call. Clients are closed by the suite's fixture. +""" + +import asyncio + +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server import ChannelMode +from ably.pubsub.server.liveobjects import PathObject +from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until +from test.uts.helpers.sandbox import random_id +from test.uts.objects.helpers.standard_test_pool import objects_channel_options + +CONNECT_TIMEOUT = 10.0 +SYNC_TIMEOUT = 15.0 + + +async def connected_client(api_key, use_binary_protocol): + """The specification's `Realtime(options: {...})`, connected and CONNECTED.""" + client = sandbox_realtime_client(api_key, auto_connect=False, use_binary_protocol=use_binary_protocol) + client.connect() + await await_connection_state(client, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + return client + + +async def synced_root(channel): + """The specification's `AWAIT channel.object.get()`, bounded at `SYNC_TIMEOUT`.""" + try: + return await asyncio.wait_for(channel.object.get(), SYNC_TIMEOUT) + except asyncio.TimeoutError: + raise AssertionError( + f'Timed out after {SYNC_TIMEOUT}s waiting for channel.object.get() on {channel.name!r}') from None + + +# UTS: objects/integration/RTO4-RTO5/attach-sync-get-0 +async def test_rto4_rto5_attach_sync_get(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-sync-' + random_id() + + client = await connected_client(realtime_sandbox.key_str, use_binary_protocol) + channel = client.channels.get(channel_name, objects_channel_options()) + + root = await synced_root(channel) + + assert isinstance(root, PathObject) + assert root.path() == '' + + +# UTS: objects/integration/RTO5-RTO17/two-clients-sync-0 +async def test_rto5_rto17_two_clients_sync(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-two-sync-' + random_id() + + client_a = await connected_client(realtime_sandbox.key_str, use_binary_protocol) + client_b = await connected_client(realtime_sandbox.key_str, use_binary_protocol) + + channel_a = client_a.channels.get(channel_name, objects_channel_options()) + channel_b = client_b.channels.get(channel_name, objects_channel_options()) + + # Client A creates data + root_a = await synced_root(channel_a) + await root_a.set('key1', 'value1') + + # Client B attaches and syncs, and should see the data + root_b = await synced_root(channel_b) + await wall_clock_poll_until( + lambda: root_b.get('key1').as_primitive().value() == 'value1', + description="client B to read key1 == 'value1'") + + assert root_b.get('key1').as_primitive().value() == 'value1' + + +# UTS: objects/integration/RTO17/reattach-resyncs-0 +async def test_rto17_reattach_resyncs(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-reattach-' + random_id() + + client = await connected_client(realtime_sandbox.key_str, use_binary_protocol) + channel = client.channels.get(channel_name, objects_channel_options()) + root = await synced_root(channel) + + # Set some data. The write is applied locally once the server ACKs it (RTO20), so + # it reads back as soon as `set` returns. + await root.set('before_detach', 'hello') + assert root.get('before_detach').as_primitive().value() == 'hello' + + # Detach and re-attach + await channel.detach() + await channel.attach() + + # Re-sync should restore data + root = await synced_root(channel) + await wall_clock_poll_until( + lambda: root.get('before_detach').as_primitive().value() == 'hello', + description="before_detach to read 'hello' after the re-sync") + + assert root.get('before_detach').as_primitive().value() == 'hello' + + +# UTS: objects/integration/RTO4/attach-subscribe-only-0 +async def test_rto4_attach_subscribe_only(realtime_sandbox, use_binary_protocol): + channel_name = 'objects-subscribe-only-' + random_id() + + client = await connected_client(realtime_sandbox.key_str, use_binary_protocol) + channel = client.channels.get(channel_name, objects_channel_options(ChannelMode.OBJECT_SUBSCRIBE)) + + root = await synced_root(channel) + + assert isinstance(root, PathObject) + assert root.size() == 0 diff --git a/test/uts/objects/integration/proxy/__init__.py b/test/uts/objects/integration/proxy/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/uts/objects/integration/proxy/conftest.py b/test/uts/objects/integration/proxy/conftest.py new file mode 100644 index 00000000..9491c7ce --- /dev/null +++ b/test/uts/objects/integration/proxy/conftest.py @@ -0,0 +1,78 @@ +"""Fixtures the LiveObjects proxy integration specification shares. + +The specification in this package runs its traffic through `uts-proxy`, so each +test opens a proxy session and closes it again afterwards. The `realtime_sandbox` +app the clients connect to comes from the parent package, which provisions it +once for the whole LiveObjects integration tier. The fixtures in +`test/uts/realtime/integration/proxy/conftest.py` are not visible from here, so +the same two are defined again; `ensure_proxy()` hands every caller the one +control process, so a run that collects both packages still starts only one. + +`test/uts/helpers/proxy.py` describes the proxy itself and the two environment +variables that change how it is obtained and started. +""" + +import os + +import pytest +import pytest_asyncio + +from test.uts.helpers.proxy import create_proxy_session, ensure_proxy, stop_proxy + +# What a test in this package gets, in seconds, in place of the 120 the parent +# package gives the rest of the tier. The first proxy test to run waits for the +# binary to be downloaded on a cold cache and for the control process to come up, +# and a fault test sits through a disconnect, a reconnection and a second objects +# sync before it can assert anything. +SUITE_TIMEOUT = 300 + +__package_dir = os.path.dirname(os.path.abspath(__file__)) + + +def pytest_collection_modifyitems(items): + # The parent package marks everything beneath it, this package included, with + # its own shorter timeout, and pytest-timeout reads the first of the item's own + # markers. Putting this one at the front is what makes it the one read, + # whichever order the two hooks happen to run in. + for item in items: + if os.path.abspath(str(item.fspath)).startswith(__package_dir + os.sep): + item.add_marker(pytest.mark.timeout(SUITE_TIMEOUT), append=False) + + +@pytest_asyncio.fixture(scope='session') +async def proxy_control(): + """The running `uts-proxy` control API, shared by every test in the package. + + One control process serves any number of sessions, each on a port of its own, + so it is started once and reaped when the run ends. `proxy_session` asks for + this fixture, so a test that opens sessions does not have to. + """ + await ensure_proxy() + yield + stop_proxy() + + +@pytest_asyncio.fixture +async def proxy_session(proxy_control): + """Opens proxy sessions, and closes every one of them when the test ends. + + This is the specification's `create_proxy_session(...)` together with its + `AFTER EACH TEST: IF session IS NOT null: session.close()`. A test calls it as + it would the function: + + session = await proxy_session(rules=[...]) + + and leaves the closing alone, so a test that fails part way through still + gives its sessions back. + """ + sessions = [] + + async def open_session(**options): + session = await create_proxy_session(**options) + sessions.append(session) + return session + + yield open_session + + for session in reversed(sessions): + await session.close() diff --git a/test/uts/objects/integration/proxy/objects_faults_test.py b/test/uts/objects/integration/proxy/objects_faults_test.py new file mode 100644 index 00000000..530cd5dd --- /dev/null +++ b/test/uts/objects/integration/proxy/objects_faults_test.py @@ -0,0 +1,451 @@ +"""Derived from uts/objects/integration/proxy/objects_faults.md in ably/specification. + +Spec points: RTO5a2, RTO7, RTO8, RTO17, RTO20e, RTO20e1 + +Each test routes a client through a `uts-proxy` session and faults the objects sync +against the real sandbox: the connection dropped on the first OBJECT_SYNC, a disconnect +while another client writes, a DETACHED injected onto an attached channel, an ATTACHED +injected to restart the sync followed by a channel ERROR while a write waits for it, and +the first OBJECT_SYNC delayed while another client writes. Only the public API is used; +the proxy's event log is the second witness where the specification reads it. + +**Clients.** A client under test points at the session (`endpoint='localhost'`, +`port=session.proxy_port`, `tls=False`, `use_binary_protocol=False`) and authenticates with +an `auth_callback` returning a locally signed Ably JWT: the session speaks plain WebSocket, +over which basic auth is refused, and a JWT signed here puts nothing in the event log. The +specification's `key: api_key` on those clients is that callback. Client A, where a test +has one, goes straight to the sandbox with the key, as the specification builds it. There +is no `## Protocol Variants` section, so every client speaks JSON, which the proxy requires +in any case. + +**Rules and the log.** A frame rule matches `action` as a string, and the proxy resolves +names only up to AUTH, so OBJECT_SYNC is matched as `'20'`. The log's frames carry +`direction` and a decoded `message` whose `action` is an integer. + +**Transient and already-held states.** The specification's `AWAIT_STATE ... DISCONNECTED` +waits for a state the client leaves at once: it reconnects immediately (RTN15a), so the +state can be gone before a wait that starts after the fault is registered. A recorder is +registered on the connection before it connects and the waits read the recorded list, +which also tells the reconnection's CONNECTED from the first one. Likewise RTO17's +`AWAIT_STATE channel.state == attached` follows a DETACHED injected onto an attached +channel, so the wait is for ATTACHING followed by ATTACHED in the recorded sequence. + +**Un-awaited calls.** `channel.attach()` and RTO20e's `pending = root.set(...)` are written +without `AWAIT`; they run as tasks. `poll_until_success` is the specification's: a read that +raises `AblyException` (the channel transiently not attached, RTO25b) means "not yet", and +the most recent such error is raised if the wait runs out. Any other error fails at once, +since it means the read is wrong rather than early. + +**Cleanup.** The specification's `AFTER EACH TEST` is the fixtures': the suite closes every +client a test built and `proxy_session` closes every session it opened. + +A primitive's `value()` is reached through `as_primitive()`, since ably-python's +`PathObject` is partitioned by type (RTTS); `root` is a `LiveMapPathObject`, so `set` is +called on it directly. +""" + +import asyncio + +import pytest + +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server import AblyException +from ably.pubsub.server.liveobjects import PathObject +from ably.pubsub.types.channelstate import ChannelState +from test.uts.helpers.client import ( + await_channel_state, + await_connection_state, + sandbox_realtime_client, + wall_clock_poll_until, +) +from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt, random_id +from test.uts.objects.helpers.standard_test_pool import objects_channel_options + +# The specification's `WITH timeout` values, as wall-clock seconds. +CONNECT_TIMEOUT = 15.0 +DISCONNECT_TIMEOUT = 15.0 +RECONNECT_TIMEOUT = 30.0 +SYNC_TIMEOUT = 15.0 +RESYNC_TIMEOUT = 30.0 +REATTACH_TIMEOUT = 30.0 +FAILED_TIMEOUT = 15.0 +PENDING_TIMEOUT = 15.0 +ACK_TIMEOUT = 10.0 +VALUE_TIMEOUT = 15.0 + +# `poll_until_success`'s defaults, as `uts/README.md` gives them. +POLL_SUCCESS_TIMEOUT = 10.0 +POLL_SUCCESS_INTERVAL = 0.5 + +# How often a recorded state list is re-read. The list is append-only, so nothing is +# missed at any interval; a short one keeps a wait from outlasting the reconnection. +STATE_POLL_INTERVAL = 0.05 + +# RTO20e's `WAIT 500ms`, for the ACKed operation to reach the wait for SYNCED. +PARK_DELAY = 0.5 + +# The actions `uts/docs/proxy.md` tabulates, as a logged frame's `message.action`. +ACK_ACTION = 1 +OBJECT_ACTION = 19 + + +def jwt_auth_callback(api_key): + """The credentials of a proxied client: an Ably JWT signed locally for `api_key`.""" + key_name = extract_key_name(api_key) + key_secret = extract_key_secret(api_key) + + async def auth_callback(params): + return generate_jwt(key_name, key_secret) + + return auth_callback + + +def proxied_client(session, api_key): + """The specification's `Realtime(options: ClientOptions(...))` pointed at `session`.""" + return sandbox_realtime_client( + auth_callback=jwt_auth_callback(api_key), + endpoint='localhost', + port=session.proxy_port, + tls=False, + use_binary_protocol=False, + auto_connect=False, + ) + + +def direct_client(api_key): + """The specification's client A, which goes straight to the sandbox with the key.""" + return sandbox_realtime_client(api_key, auto_connect=False) + + +def record_states(emitter): + """Records every state `emitter` enters from now on, in order.""" + states = [] + + def on_state_change(change): + states.append(change.current) + + emitter.on(on_state_change) + return states + + +async def await_recorded_state(states, state, count=1, timeout=CONNECT_TIMEOUT): + """This file's `AWAIT_STATE` for a connection: waits until `states` holds `count` entries of `state`.""" + try: + await wall_clock_poll_until( + lambda: states.count(state) >= count, + timeout=timeout, + description=f'{count} state change(s) to {state.value}', + interval=STATE_POLL_INTERVAL, + ) + except AssertionError as error: + recorded = [recorded_state.value for recorded_state in states] + raise AssertionError(f'{error}; the states recorded were {recorded}') from None + + +def contains_in_order(recorded, expected): + """The specifications' `CONTAINS_IN_ORDER`: `expected` as a subsequence of `recorded`.""" + remaining = list(expected) + for item in recorded: + if remaining and item == remaining[0]: + remaining.pop(0) + return not remaining + + +async def bounded(awaitable, timeout, description): + """An `AWAIT ... WITH timeout`, failing with `description` when the time runs out.""" + try: + return await asyncio.wait_for(awaitable, timeout) + except asyncio.TimeoutError: + raise AssertionError(f'Timed out after {timeout}s waiting for {description}') from None + + +def in_background(coroutine): + """Starts `coroutine` without awaiting it, as an un-awaited call in the specification. + + What it ends with is retrieved when it ends, so an outcome the specification does not + assert on is not reported again as an exception nobody retrieved. + """ + task = asyncio.ensure_future(coroutine) + task.add_done_callback(lambda done: done.cancelled() or done.exception()) + return task + + +async def poll_until_success(condition, timeout=POLL_SUCCESS_TIMEOUT, description='condition'): + """The specification's `poll_until_success`, for a read across a fault and its recovery.""" + loop = asyncio.get_running_loop() + deadline = loop.time() + timeout + last_error = None + while True: + try: + result = condition() + if result: + return result + except AblyException as error: + last_error = error + if loop.time() >= deadline: + if last_error is not None: + raise last_error + raise AssertionError(f'Timed out after {timeout}s waiting for {description}') + await asyncio.sleep(POLL_SUCCESS_INTERVAL) + + +def object_publish_acked(log): + """Whether the log holds the server's ACK for an OBJECT frame the client sent. + + The specification polls for any server-to-client ACK; the client sends nothing else + that is ACKed before the `set`, so requiring the OBJECT frame first names the same + frame and cannot be satisfied by anything else. + """ + published = False + for event in log: + if event['type'] != 'ws_frame': + continue + action = (event.get('message') or {}).get('action') + if event.get('direction') == 'client_to_server' and action == OBJECT_ACTION: + published = True + elif published and event.get('direction') == 'server_to_client' and action == ACK_ACTION: + return True + return False + + +# UTS: objects/proxy/RTO5a2-RTO17/sync-interrupted-reconnect-0 +async def test_rto5a2_rto17_sync_interrupted_reconnect(realtime_sandbox, proxy_session): + channel_name = 'objects-sync-interrupt-' + random_id() + + # Disconnect after first OBJECT_SYNC frame + session = await proxy_session(rules=[{ + 'match': {'type': 'ws_frame_to_client', 'action': '20'}, + 'action': {'type': 'disconnect'}, + 'times': 1, + 'comment': 'RTO5a2: Disconnect after first OBJECT_SYNC to interrupt sync', + }]) + + client = proxied_client(session, realtime_sandbox.key_str) + channel = client.channels.get(channel_name, objects_channel_options()) + states = record_states(client.connection) + + client.connect() + await await_recorded_state(states, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + + # First attach triggers sync; proxy disconnects mid-sync + in_background(channel.attach()) + await await_recorded_state(states, ConnectionState.DISCONNECTED, timeout=DISCONNECT_TIMEOUT) + + # Client auto-reconnects; re-attach triggers fresh sync + await await_recorded_state(states, ConnectionState.CONNECTED, count=2, timeout=RECONNECT_TIMEOUT) + + # get() waits for SYNCED, and will only resolve if the re-sync completes + root = await bounded(channel.object.get(), RESYNC_TIMEOUT, 'channel.object.get() after the re-sync') + + assert isinstance(root, PathObject) + assert root.path() == '' + + +# UTS: objects/proxy/RTO7-RTO8/mutations-buffered-during-resync-0 +async def test_rto7_rto8_mutations_buffered_during_resync(realtime_sandbox, proxy_session): + channel_name = 'objects-buffer-resync-' + random_id() + + # Client A: direct connection (no proxy), publishes mutations + client_a = direct_client(realtime_sandbox.key_str) + client_a.connect() + await await_connection_state(client_a, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + + channel_a = client_a.channels.get(channel_name, objects_channel_options()) + root_a = await bounded(channel_a.object.get(), SYNC_TIMEOUT, "client A's channel.object.get()") + + # Set initial data + await root_a.set('key1', 'initial') + + # Client B: through proxy, will be disconnected + session = await proxy_session(rules=[]) + + client_b = proxied_client(session, realtime_sandbox.key_str) + channel_b = client_b.channels.get(channel_name, objects_channel_options()) + states_b = record_states(client_b.connection) + + # Client B connects and syncs + client_b.connect() + await await_recorded_state(states_b, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + + root_b = await bounded(channel_b.object.get(), SYNC_TIMEOUT, "client B's channel.object.get()") + await poll_until_success( + lambda: root_b.get('key1').as_primitive().value() == 'initial', + description="client B to read key1 == 'initial'") + + # Disconnect client B + await session.trigger_action({'type': 'disconnect'}) + await await_recorded_state(states_b, ConnectionState.DISCONNECTED, timeout=DISCONNECT_TIMEOUT) + + # While B is disconnected, A publishes a mutation + await root_a.set('key1', 'updated_during_disconnect') + + # Client B reconnects and re-syncs; the mutation should be visible + await await_recorded_state(states_b, ConnectionState.CONNECTED, count=2, timeout=RECONNECT_TIMEOUT) + + root_b = await bounded( + channel_b.object.get(), SYNC_TIMEOUT, "client B's channel.object.get() after reconnecting") + await poll_until_success( + lambda: root_b.get('key1').as_primitive().value() == 'updated_during_disconnect', + timeout=VALUE_TIMEOUT, + description="client B to read key1 == 'updated_during_disconnect'") + + assert root_b.get('key1').as_primitive().value() == 'updated_during_disconnect' + + +# UTS: objects/proxy/RTO17/server-detach-resync-0 +async def test_rto17_server_detach_resync(realtime_sandbox, proxy_session): + channel_name = 'objects-detach-resync-' + random_id() + + session = await proxy_session(rules=[]) + + client = proxied_client(session, realtime_sandbox.key_str) + channel = client.channels.get(channel_name, objects_channel_options()) + + client.connect() + await await_connection_state(client, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + + root = await bounded(channel.object.get(), SYNC_TIMEOUT, 'channel.object.get()') + + # Set some data. The write is applied locally once the server ACKs it (RTO20), so it + # reads back as soon as `set` returns. + await root.set('before_detach', 'hello') + assert root.get('before_detach').as_primitive().value() == 'hello' + + channel_states = record_states(channel) + + # Inject server-initiated DETACHED + await session.trigger_action({ + 'type': 'inject_to_client', + 'message': { + 'action': 13, + 'channel': channel_name, + }, + }) + + # Client should auto-re-attach (RTL13a). The channel is ATTACHED when the wait begins, + # so it is the recorded re-attach that is waited for. + try: + await wall_clock_poll_until( + lambda: contains_in_order(channel_states, [ChannelState.ATTACHING, ChannelState.ATTACHED]), + timeout=REATTACH_TIMEOUT, + description='the channel to re-attach after the injected DETACHED', + interval=STATE_POLL_INTERVAL) + except AssertionError as error: + recorded = [recorded_state.value for recorded_state in channel_states] + raise AssertionError(f'{error}; the channel states recorded were {recorded}') from None + assert channel.state == ChannelState.ATTACHED + + # Re-sync should restore data + root = await bounded(channel.object.get(), SYNC_TIMEOUT, 'channel.object.get() after the re-attach') + await poll_until_success( + lambda: root.get('before_detach').as_primitive().value() == 'hello', + timeout=VALUE_TIMEOUT, + description="before_detach to read 'hello' after the re-sync") + + assert root.get('before_detach').as_primitive().value() == 'hello' + + +# UTS: objects/proxy/RTO20e/publish-fails-on-channel-failed-0 +async def test_rto20e_publish_fails_on_channel_failed(realtime_sandbox, proxy_session): + channel_name = 'objects-publish-failed-' + random_id() + + session = await proxy_session(rules=[]) + + client = proxied_client(session, realtime_sandbox.key_str) + channel = client.channels.get(channel_name, objects_channel_options()) + + client.connect() + await await_connection_state(client, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + + root = await bounded(channel.object.get(), SYNC_TIMEOUT, 'channel.object.get()') + + # Force the objects back into SYNCING: inject an ATTACHED (action 11) carrying the + # HAS_OBJECTS flag (bit 7, i.e. flags: 128). RTO4c starts a new sync sequence on every + # ATTACHED protocol message; the server never sent this ATTACHED, so no OBJECT_SYNC + # follows and the objects remain SYNCING. The channel itself stays ATTACHED. + await session.trigger_action({ + 'type': 'inject_to_client', + 'message': {'action': 11, 'channel': channel_name, 'flags': 128}, + }) + + # Mutate WHILE SYNCING: the channel is ATTACHED so the write preconditions (RTO26) + # pass and the publish + ACK complete against the real server; publishAndApply then + # waits for a SYNCED that will never arrive (RTO20e). Do not await yet. + pending = asyncio.ensure_future(root.set('key', 'value')) + + # Ensure the operation is in the RTO20e sync-wait, not still publishing: wait until + # the proxy log shows the server's ACK for the OBJECT publish, then allow a brief + # real-time yield for the client to move the ACKed operation into the wait. + async def acked(): + return object_publish_acked(await session.get_log()) + + await wall_clock_poll_until(acked, timeout=ACK_TIMEOUT, description='the server to ACK the OBJECT publish') + await asyncio.sleep(PARK_DELAY) + + # The channel enters FAILED whilst the operation waits for SYNCED (RTO20e1) + await session.trigger_action({ + 'type': 'inject_to_client', + 'message': { + 'action': 9, + 'channel': channel_name, + 'error': {'statusCode': 400, 'code': 90000, 'message': 'injected error'}, + }, + }) + + await await_channel_state(channel, ChannelState.FAILED, timeout=FAILED_TIMEOUT) + + with pytest.raises(AblyException) as excinfo: + await bounded(pending, PENDING_TIMEOUT, 'the pending set() to fail') + + error = excinfo.value + assert error.code == 92008 + assert error.status_code == 400 + # RTO20e1: cause is set to RealtimeChannel.errorReason, the injected channel ERROR + assert error.cause is not None + assert error.cause.code == 90000 + + +# UTS: objects/proxy/RTO5-RTO7/publish-during-sync-echo-after-0 +async def test_rto5_rto7_publish_during_sync_echo_after(realtime_sandbox, proxy_session): + channel_name = 'objects-publish-during-sync-' + random_id() + + # Client A: direct, no proxy + client_a = direct_client(realtime_sandbox.key_str) + client_a.connect() + await await_connection_state(client_a, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + + channel_a = client_a.channels.get(channel_name, objects_channel_options()) + root_a = await bounded(channel_a.object.get(), SYNC_TIMEOUT, "client A's channel.object.get()") + + # Set up initial data + await root_a.set('existing', 'before') + + # Client B: through proxy with delayed OBJECT_SYNC + session = await proxy_session(rules=[{ + 'match': {'type': 'ws_frame_to_client', 'action': '20'}, + 'action': {'type': 'delay', 'delayMs': 3000}, + 'times': 1, + 'comment': 'Delay first OBJECT_SYNC to keep B in SYNCING state', + }]) + + client_b = proxied_client(session, realtime_sandbox.key_str) + channel_b = client_b.channels.get(channel_name, objects_channel_options()) + + # Start client B, which stays SYNCING while its first OBJECT_SYNC is delayed + client_b.connect() + await await_connection_state(client_b, ConnectionState.CONNECTED, timeout=CONNECT_TIMEOUT) + in_background(channel_b.attach()) + + # While B is syncing, A publishes a mutation + await root_a.set('existing', 'after') + + # B's get() will resolve once the delayed sync completes + root_b = await bounded(channel_b.object.get(), RESYNC_TIMEOUT, "client B's channel.object.get()") + + # The mutation from A should be visible (either in sync data or buffered OBJECT) + await poll_until_success( + lambda: root_b.get('existing').as_primitive().value() == 'after', + timeout=VALUE_TIMEOUT, + description="client B to read existing == 'after'") + + assert root_b.get('existing').as_primitive().value() == 'after' diff --git a/test/uts/objects/unit/__init__.py b/test/uts/objects/unit/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/test/uts/objects/unit/instance_test.py b/test/uts/objects/unit/instance_test.py new file mode 100644 index 00000000..1cdefc80 --- /dev/null +++ b/test/uts/objects/unit/instance_test.py @@ -0,0 +1,356 @@ +"""Derived from uts/objects/unit/instance.md in ably/specification. + +Spec points: RTINS3, RTINS4, RTINS5, RTINS6, RTINS9, RTINS10, RTINS12, RTINS13, RTINS14, +RTINS15, RTINS16, RTTS8a, RTTS9d, RTTS10 + +The specification is written against the untyped `Instance`, one class carrying every +method. ably-python partitions it per RTTS7-RTTS10: `id` and `type` are properties, `get`, +`compact_json` and the three view helpers are on the base `Instance`, and a type-specific method +is reached through the checked view helper for the wrapped type, so `inst.value()` on a +counter is `inst.as_live_counter().value()` and `inst.set(k, v)` on a map is +`await inst.as_live_map().set(k, v)`. A view helper asked for a type the instance does not +wrap raises AblyException 92007 (RTTS9d). + +That makes three of the specification's reads unreachable: `value()` on a map instance +(RTINS4d), `size()` on a counter instance (RTINS9c) and `subscribe` on a primitive instance +(RTINS16c), since the typed instance for those types has no such method and the views +that do have it refuse the wrapped type. Those tests assert the wrapped `type`, the +absence of the method, and the 92007 the view helpers raise in its place (S-5). + +The snapshot is `compact_json()` alone (RTTS7d), so RTINS10's test takes it there (S-6). + +`setup_synced_channel` uses the standard mock, which ACKs every OBJECT the client sends, +so each write is awaited to completion and is applied locally by the time it returns +(RTO20). +""" + +import pytest + +from ably.pubsub.liveobjects import publicmessage +from ably.pubsub.liveobjects.enums import ValueType +from ably.pubsub.liveobjects.instance import Instance, InstanceSubscriptionEvent, PrimitiveInstance +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction +from ably.pubsub.liveobjects.subscription import Subscription +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import poll_until +from test.uts.helpers.clock import settle +from test.uts.objects.helpers.standard_test_pool import ( + assert_unchanged_after_quiescence, + build_counter_inc, + build_map_set, + build_object_message, + build_public_object_message, + remote_serial, + setup_synced_channel, +) + + +def assert_view_refused(view_helper): + """Asserts that calling an `Instance` view helper fails fast with 92007 (RTTS9d).""" + with pytest.raises(AblyException) as excinfo: + view_helper() + assert excinfo.value.code == 92007 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTINS3/id-returns-objectid-0 +async def test_rtins3_id_returns_objectid(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + counter_inst = root.get('score').instance() + assert counter_inst.id == 'counter:score@1000' + + map_inst = root.get('profile').instance() + assert map_inst.id == 'map:profile@1000' + + +# UTS: objects/unit/RTINS4/value-counter-0 +async def test_rtins4_value_counter(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + counter_inst = root.get('score').instance() + assert counter_inst.as_live_counter().value() == 100 + + map_inst = root.instance() + # S-5: the specification asserts `map_inst.value() == null` (RTINS4d). A map instance has + # no `value()` (RTTS10a), and the two views that do have one refuse a map (RTTS9d) + assert map_inst.type is ValueType.LIVE_MAP + assert not hasattr(map_inst, 'value') + assert_view_refused(map_inst.as_primitive) + assert_view_refused(map_inst.as_live_counter) + + +# UTS: objects/unit/RTINS5/get-wraps-entry-0 +async def test_rtins5_get_wraps_entry(): + client, channel, root, mock_ws = await setup_synced_channel('test') + root_inst = root.instance() + + name_inst = root_inst.get('name') + assert isinstance(name_inst, Instance) + assert name_inst.as_primitive().value() == 'Alice' + + score_inst = root_inst.get('score') + assert score_inst.id == 'counter:score@1000' + + null_inst = root_inst.get('nonexistent') + assert null_inst is None + + +# UTS: objects/unit/RTINS6/entries-yields-instances-0 +async def test_rtins6_entries_yields_instances(): + client, channel, root, mock_ws = await setup_synced_channel('test') + root_inst = root.instance() + + entries = {} + for key, inst in root_inst.as_live_map().entries(): + entries[key] = inst + + assert len(entries) == 7 + assert isinstance(entries['name'], Instance) + assert entries['name'].as_primitive().value() == 'Alice' + + +# UTS: objects/unit/RTINS9/size-0 +async def test_rtins9_size(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + root_inst = root.instance() + assert root_inst.as_live_map().size() == 7 + + counter_inst = root.get('score').instance() + # S-5: the specification asserts `counter_inst.size() == null` (RTINS9c). A counter + # instance has no `size()` (RTTS10b), and the map view that has one refuses a counter + # (RTTS9d) + assert counter_inst.type is ValueType.LIVE_COUNTER + assert not hasattr(counter_inst, 'size') + assert_view_refused(counter_inst.as_live_map) + + +# UTS: objects/unit/RTINS10/compact-0 +async def test_rtins10_compact(): + client, channel, root, mock_ws = await setup_synced_channel('test') + root_inst = root.instance() + + # S-6: the specification calls `compact()`, which reads the same for these values + result = root_inst.compact_json() + + assert result['name'] == 'Alice' + assert result['score'] == 100 + assert result['profile']['email'] == 'alice@example.com' + + +# UTS: objects/unit/RTINS12/set-delegates-0 +async def test_rtins12_set_delegates(): + client, channel, root, mock_ws = await setup_synced_channel('test') + root_inst = root.instance() + + await root_inst.as_live_map().set('name', 'Bob') + + assert root.get('name').as_primitive().value() == 'Bob' + + +# UTS: objects/unit/RTINS12d/set-non-map-throws-0 +async def test_rtins12d_set_non_map_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + + # `set` is reached through the map view, which refuses a counter with the 92007 and + # status 400 that RTINS12d requires of `set` (RTTS9d) + with pytest.raises(AblyException) as excinfo: + await counter_inst.as_live_map().set('key', 'value') + + assert excinfo.value.code == 92007 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTINS13/remove-delegates-0 +async def test_rtins13_remove_delegates(): + client, channel, root, mock_ws = await setup_synced_channel('test') + root_inst = root.instance() + assert root.get('name').as_primitive().value() == 'Alice' + + await root_inst.as_live_map().remove('name') + + assert root.get('name').as_primitive().value() is None + + +# UTS: objects/unit/RTINS14/increment-delegates-0 +async def test_rtins14_increment_delegates(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + + await counter_inst.as_live_counter().increment(25) + + assert root.get('score').as_live_counter().value() == 125 + + +# UTS: objects/unit/RTINS14d/increment-non-counter-throws-0 +async def test_rtins14d_increment_non_counter_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + map_inst = root.instance() + + # `increment` is reached through the counter view, which refuses a map with the 92007 + # and status 400 that RTINS14d requires of `increment` (RTTS9d) + with pytest.raises(AblyException) as excinfo: + await map_inst.as_live_counter().increment(5) + + assert excinfo.value.code == 92007 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTINS15/decrement-delegates-0 +async def test_rtins15_decrement_delegates(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + + await counter_inst.as_live_counter().decrement(10) + + assert root.get('score').as_live_counter().value() == 90 + + +# UTS: objects/unit/RTINS14a/increment-default-0 +async def test_rtins14a_increment_default(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + + await counter_inst.as_live_counter().increment() + + assert root.get('score').as_live_counter().value() == 101 + + +# UTS: objects/unit/RTINS15a/decrement-default-0 +async def test_rtins15a_decrement_default(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + + await counter_inst.as_live_counter().decrement() + + assert root.get('score').as_live_counter().value() == 99 + + +# UTS: objects/unit/RTINS16/subscribe-receives-events-0 +async def test_rtins16_subscribe_receives_events(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + events = [] + sub = counter_inst.as_live_counter().subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, timeout=5, description='the listener to receive the COUNTER_INC') + await settle() + + assert isinstance(sub, Subscription) + assert len(events) == 1 + assert isinstance(events[0], InstanceSubscriptionEvent) + assert isinstance(events[0].object, Instance) + assert events[0].object.id == 'counter:score@1000' + + +# UTS: objects/unit/RTINS16c/subscribe-primitive-throws-0 +async def test_rtins16c_subscribe_primitive_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + name_inst = root.instance().get('name') + + # S-5: the specification calls `name_inst.subscribe(...)` and expects 92007 (RTINS16c). + # A primitive instance has no `subscribe` (RTTS7b, RTTS10c), and the two views that do + # have one refuse a primitive with 92007 (RTTS9d) + assert isinstance(name_inst, PrimitiveInstance) + assert name_inst.type is ValueType.STRING + assert not hasattr(name_inst, 'subscribe') + assert_view_refused(name_inst.as_live_map) + assert_view_refused(name_inst.as_live_counter) + + +# UTS: objects/unit/RTINS16e2/subscription-event-message-0 +async def test_rtins16e2_subscription_event_message(): + client, channel, root, mock_ws = await setup_synced_channel('test') + root_inst = root.instance() + events = [] + root_inst.as_live_map().subscribe(events.append) + + map_set = build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote') + mock_ws.send_to_client(build_object_message('test', [map_set])) + await poll_until(lambda: len(events) >= 1, timeout=5, description='the listener to receive the MAP_SET') + + assert isinstance(events[0].object, Instance) + assert events[0].object.id == 'root' + assert events[0].message is not None + assert isinstance(events[0].message, publicmessage.ObjectMessage) + assert events[0].message.channel == 'test' + assert events[0].message.operation.action == ObjectOperationAction.MAP_SET + assert events[0].message.operation.object_id == 'root' + assert events[0].message.operation.map_set.key == 'name' + # The whole message is the one PAOM3 derives from the MAP_SET as it arrived + assert events[0].message == build_public_object_message(map_set, 'test') + + +# UTS: objects/unit/RTINS16f/subscribe-returns-subscription-0 +async def test_rtins16f_subscribe_returns_subscription(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + events = [] + sub = counter_inst.as_live_counter().subscribe(events.append) + sub.unsubscribe() + + # Quiescence control: a listener still subscribed to the same counter, which the + # same dispatch reaches + control_events = [] + counter_inst.as_live_counter().subscribe(control_events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote'), + ])) + + await assert_unchanged_after_quiescence( + lambda: len(events), lambda: len(control_events) >= 1, + description='the control listener receives the COUNTER_INC') + assert len(events) == 0 + + +# UTS: objects/unit/RTINS16g/subscription-follows-identity-0 +async def test_rtins16g_subscription_follows_identity(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + events = [] + counter_inst.as_live_counter().subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'score', {'objectId': 'counter:new@2000'}, remote_serial(0), 'remote'), + ])) + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 10, '100', 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, timeout=5, description='the listener to receive the COUNTER_INC') + + assert len(events) >= 1 + # RTINS16e1: the delivered event carries the Instance wrapping the object that fired, + # so the assertion reads the event rather than the `counter_inst` handle + assert isinstance(events[0].object, Instance) + assert events[0].object.id == 'counter:score@1000' + # The frames are processed in order, so `score` already pointed at the new counter + # when the original one was incremented, and the event is that increment + assert root.get('score').instance().id == 'counter:new@2000' + assert events[0].message.operation.counter_inc.number == 10 + assert events[0].object.as_live_counter().value() == 110 + + +# UTS: objects/unit/RTINS16h/subscribe-no-side-effects-0 +async def test_rtins16h_subscribe_no_side_effects(): + client, channel, root, mock_ws = await setup_synced_channel('test') + counter_inst = root.get('score').instance() + channel_state_before = channel.state + sent_before = len(mock_ws.messages_from_client) + assert channel_state_before == ChannelState.ATTACHED + + sub = counter_inst.as_live_counter().subscribe(lambda event: None) + await settle() + + assert isinstance(sub, Subscription) + assert channel.state == channel_state_before + # Nothing was sent to the server either, such as an attach or a detach + assert len(mock_ws.messages_from_client) == sent_before diff --git a/test/uts/objects/unit/internal_live_counter_api_test.py b/test/uts/objects/unit/internal_live_counter_api_test.py new file mode 100644 index 00000000..088c7202 --- /dev/null +++ b/test/uts/objects/unit/internal_live_counter_api_test.py @@ -0,0 +1,142 @@ +"""Derived from uts/objects/unit/internal_live_counter_api.md in ably/specification. + +Spec points: RTLC5, RTLC5c, RTLC11, RTLC11b1, RTLC12, RTLC12e1, RTLC12e2, RTLC12e3, RTLC12e5, +RTLC12g, RTLC13, RTLC13b + +The specification reaches `InternalLiveCounter` through the untyped `PathObject` and +`Instance`; ably-python partitions those by type (RTTS), so a counter's `value`, +`increment`, `decrement` and `subscribe` are read through `as_live_counter()`. + +The tests that capture what the client sends use the standard synced-channel mock, which +is the specification's hand-written one: it records each OBJECT message and then ACKs it +with `ack_serial(msgSerial, i)`, so every awaited write resolves (RTO20). Captured messages +are the decoded JSON wire, and each captured operation is also compared with the whole +protocol v6 operation, so a legacy (v5) payload field fails the test. +""" + +import pytest + +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import poll_until +from test.uts.objects.helpers.standard_test_pool import ( + build_counter_inc, + build_object_message, + objects_connected_message, + setup_synced_channel, + standard_mock_websocket, +) + + +def _capturing_mock_websocket(captured): + """The specification's capturing mock: the standard synced-channel conversation, with each + OBJECT message the client sends appended to `captured` before it is ACKed.""" + return standard_mock_websocket( + on_object=captured.append, + connected=objects_connected_message(connection_key='key-1'), + ) + + +# UTS: objects/unit/RTLC5/value-returns-data-0 +async def test_rtlc5_value_returns_data(): + ctx = await setup_synced_channel('test') + + counter = ctx.root.get('score') + assert counter.as_live_counter().value() == 100 + + +# UTS: objects/unit/RTLC12/increment-sends-counter-inc-0 +async def test_rtlc12_increment_sends_counter_inc(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.get('score').as_live_counter().increment(25) + + assert len(captured) == 1 + obj_msg = captured[0]['state'][0] + assert obj_msg['operation']['action'] == ObjectOperationAction.COUNTER_INC + assert obj_msg['operation']['objectId'] == 'counter:score@1000' + assert obj_msg['operation']['counterInc']['number'] == 25 + # The whole operation is the v6 wire shape: a numeric action and a `counterInc` payload + assert obj_msg['operation'] == { + 'action': int(ObjectOperationAction.COUNTER_INC), + 'objectId': 'counter:score@1000', + 'counterInc': {'number': 25}, + } + + +# UTS: objects/unit/RTLC12/increment-applies-locally-0 +async def test_rtlc12_increment_applies_locally(): + ctx = await setup_synced_channel('test') + + await ctx.root.get('score').as_live_counter().increment(50) + + assert ctx.root.get('score').as_live_counter().value() == 150 + + +# UTS: objects/unit/RTLC12e1/increment-non-number-0 +async def test_rtlc12e1_increment_non_number(): + ctx = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + await ctx.root.get('score').as_live_counter().increment('not_a_number') + + assert excinfo.value.code == 40003 + + +# UTS: objects/unit/RTLC13/decrement-negates-0 +async def test_rtlc13_decrement_negates(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.get('score').as_live_counter().decrement(15) + + assert captured[0]['state'][0]['operation']['counterInc']['number'] == -15 + assert ctx.root.get('score').as_live_counter().value() == 85 + assert captured[0]['state'][0]['operation'] == { + 'action': int(ObjectOperationAction.COUNTER_INC), + 'objectId': 'counter:score@1000', + 'counterInc': {'number': -15}, + } + + +# UTS: objects/unit/RTLC11/counter-update-on-inc-0 +async def test_rtlc11_counter_update_on_inc(): + ctx = await setup_synced_channel('test') + + updates = [] + instance = ctx.root.get('score').instance() + instance.as_live_counter().subscribe(lambda event: updates.append(event)) + + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote-site'), + ])) + + await poll_until(lambda: len(updates) >= 1, timeout=5, description='the counter update to be delivered') + + assert updates[0].message.operation.counter_inc.number == 7 + + +# The `null` row applies: in Python `None` is distinguishable from an omitted amount, which +# defaults to 1. `True` is an `int` in Python and must still be rejected as a non-number. +INVALID_AMOUNTS = [ + pytest.param(None, id='null'), + pytest.param(float('nan'), id='NaN'), + pytest.param(float('inf'), id='Infinity'), + pytest.param(float('-inf'), id='-Infinity'), + pytest.param('10', id='string'), + pytest.param(True, id='boolean'), + pytest.param([1, 2], id='array'), + pytest.param({'n': 1}, id='object'), +] + + +# UTS: objects/unit/RTLC12e1/increment-invalid-amounts-table-0 +@pytest.mark.parametrize('amount', INVALID_AMOUNTS) +async def test_rtlc12e1_increment_invalid_amounts_table(amount): + ctx = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + await ctx.root.get('score').as_live_counter().increment(amount) + + assert excinfo.value.code == 40003 diff --git a/test/uts/objects/unit/internal_live_counter_test.py b/test/uts/objects/unit/internal_live_counter_test.py new file mode 100644 index 00000000..252df270 --- /dev/null +++ b/test/uts/objects/unit/internal_live_counter_test.py @@ -0,0 +1,471 @@ +"""Derived from uts/objects/unit/internal_live_counter.md in ably/specification. + +Spec points: RTLC1, RTLC3, RTLC4, RTLC6, RTLC7, RTLC8, RTLC9, RTLC14, RTLC16, RTLO3, RTLO4a, +RTLO4b4d, RTLO4b4e, RTLO4e, RTLO5, RTLO6 + +The specification drives an `InternalLiveCounter` directly and reads the update from +`update = counter.applyOperation(...)`, or `false` when the operation is rejected. +`apply_operation` returns whether the operation applied (RTLC7g) and emits the update +through `notify_updated` (RTLC7d1a, RTLC7d5a, RTLC7d4c), so each test records the +emitted updates with `capture_updates` (shape deviation S-1): where the specification +reads `update`, the operation applied and emitted exactly one update, a no-op included; +where it reads `result == false`, the operation did not apply and emitted nothing. +`replace_data` returns its update (RTLC6h), as the specification has it. +""" + +from ably.pubsub.liveobjects.enums import ObjectsOperationSource +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction +from test.uts.helpers.clock import FakeClock +from test.uts.objects.helpers.standard_test_pool import ( + build_counter_create, + build_counter_inc, + build_map_set, + build_object_delete, + build_object_state, + capture_updates, + object_message, +) + +COUNTER_ID = 'counter:abc@1000' + +CHANNEL = ObjectsOperationSource.CHANNEL +LOCAL = ObjectsOperationSource.LOCAL + + +def _emitted_update(applied, updates): + """S-1: the specification's `update = counter.applyOperation(...)`. + + An operation that applies returns True and emits exactly one update, a no-op included. + """ + assert applied is True, 'expected the operation to apply' + assert len(updates) == 1, f'expected the operation to emit exactly one update, got {updates!r}' + return updates[0] + + +def _assert_not_applied(applied, updates): + """S-1: the specification's `result == false`: the operation did not apply and emitted nothing.""" + assert applied is False + assert updates == [] + + +# UTS: objects/unit/RTLC4/zero-value-0 +def test_rtlc4_zero_value(): + counter = InternalLiveCounter(COUNTER_ID) + + assert counter.data == 0 + assert counter.object_id == COUNTER_ID + assert counter.is_tombstone is False + assert counter.tombstoned_at is None + assert counter.create_operation_is_merged is False + assert counter.site_timeserials == {} + + +# UTS: objects/unit/RTLC9/counter-inc-basic-0 +def test_rtlc9_counter_inc_basic(): + counter = InternalLiveCounter(COUNTER_ID) + updates = capture_updates(counter) + + msg = object_message(build_counter_inc(COUNTER_ID, 5, '01', 'site1')) + applied = counter.apply_operation(msg, CHANNEL) + + assert counter.data == 5 + update = _emitted_update(applied, updates) + assert update.noop is False + assert update.update.amount == 5 + assert update.object_message is msg + + +# UTS: objects/unit/RTLC9/counter-inc-negative-0 +def test_rtlc9_counter_inc_negative(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 10 + counter.site_timeserials = {'site1': '00'} + updates = capture_updates(counter) + + msg = object_message(build_counter_inc(COUNTER_ID, -3, '01', 'site1')) + applied = counter.apply_operation(msg, CHANNEL) + + assert counter.data == 7 + update = _emitted_update(applied, updates) + assert update.update.amount == -3 + assert update.object_message is msg + + +# UTS: objects/unit/RTLC9/counter-inc-missing-number-0 +def test_rtlc9_counter_inc_missing_number(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 10 + updates = capture_updates(counter) + + msg = object_message({ + 'serial': '01', + 'siteCode': 'site1', + 'operation': { + 'action': int(ObjectOperationAction.COUNTER_INC), + 'objectId': COUNTER_ID, + 'counterInc': {}, + }, + }) + applied = counter.apply_operation(msg, CHANNEL) + + assert counter.data == 10 + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLC9/counter-inc-accumulate-0 +def test_rtlc9_counter_inc_accumulate(): + counter = InternalLiveCounter(COUNTER_ID) + + counter.apply_operation(object_message(build_counter_inc(COUNTER_ID, 10, '01', 'site1')), CHANNEL) + counter.apply_operation(object_message(build_counter_inc(COUNTER_ID, 20, '02', 'site1')), CHANNEL) + counter.apply_operation(object_message(build_counter_inc(COUNTER_ID, -5, '01', 'site2')), CHANNEL) + + assert counter.data == 25 + + +# UTS: objects/unit/RTLC8/counter-create-merge-0 +def test_rtlc8_counter_create_merge(): + counter = InternalLiveCounter(COUNTER_ID) + updates = capture_updates(counter) + + msg = object_message(build_counter_create(COUNTER_ID, {'count': 42}, '01', 'site1')) + applied = counter.apply_operation(msg, CHANNEL) + + assert counter.data == 42 + assert counter.create_operation_is_merged is True + update = _emitted_update(applied, updates) + assert update.update.amount == 42 + assert update.object_message is msg + + +# UTS: objects/unit/RTLC8/counter-create-already-merged-0 +def test_rtlc8_counter_create_already_merged(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 42 + counter.create_operation_is_merged = True + counter.site_timeserials = {'site1': '00'} + updates = capture_updates(counter) + + msg = object_message(build_counter_create(COUNTER_ID, {'count': 99}, '01', 'site1')) + applied = counter.apply_operation(msg, CHANNEL) + + assert counter.data == 42 + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLC16/counter-create-no-count-0 +def test_rtlc16_counter_create_no_count(): + counter = InternalLiveCounter(COUNTER_ID) + updates = capture_updates(counter) + + msg = object_message(build_counter_create(COUNTER_ID, {}, '01', 'site1')) + applied = counter.apply_operation(msg, CHANNEL) + + assert counter.data == 0 + assert counter.create_operation_is_merged is True + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLO4a/apply-empty-site-serial-0 +def test_rtlo4a_apply_empty_site_serial(): + counter = InternalLiveCounter(COUNTER_ID) + + msg = object_message(build_counter_inc(COUNTER_ID, 5, '01', 'site1')) + result = counter.apply_operation(msg, CHANNEL) + + # S-1: the specification's `result IS NOT false` is a True return + assert result is True + assert counter.data == 5 + + +# UTS: objects/unit/RTLO4a/reject-stale-serial-0 +def test_rtlo4a_reject_stale_serial(): + counter = InternalLiveCounter(COUNTER_ID) + counter.site_timeserials = {'site1': '05'} + counter.data = 10 + updates = capture_updates(counter) + + msg = object_message(build_counter_inc(COUNTER_ID, 99, '03', 'site1')) + result = counter.apply_operation(msg, CHANNEL) + + _assert_not_applied(result, updates) + assert counter.data == 10 + + +# UTS: objects/unit/RTLO4a/reject-equal-serial-0 +def test_rtlo4a_reject_equal_serial(): + counter = InternalLiveCounter(COUNTER_ID) + counter.site_timeserials = {'site1': '05'} + counter.data = 10 + updates = capture_updates(counter) + + msg = object_message(build_counter_inc(COUNTER_ID, 99, '05', 'site1')) + result = counter.apply_operation(msg, CHANNEL) + + _assert_not_applied(result, updates) + assert counter.data == 10 + + +# UTS: objects/unit/RTLO4a/warn-invalid-serial-0 +def test_rtlo4a_warn_invalid_serial(): + counter = InternalLiveCounter(COUNTER_ID) + updates = capture_updates(counter) + + msg_no_serial = object_message(build_counter_inc(COUNTER_ID, 5, '', 'site1')) + result1 = counter.apply_operation(msg_no_serial, CHANNEL) + + msg_no_site = object_message(build_counter_inc(COUNTER_ID, 5, '01', '')) + result2 = counter.apply_operation(msg_no_site, CHANNEL) + + assert counter.data == 0 + assert result1 is False + assert result2 is False + # S-1: neither rejected operation emitted an update + assert updates == [] + + +# UTS: objects/unit/RTLC7c/channel-source-updates-serials-0 +def test_rtlc7c_channel_source_updates_serials(): + counter = InternalLiveCounter(COUNTER_ID) + + msg = object_message(build_counter_inc(COUNTER_ID, 5, '01', 'site1')) + counter.apply_operation(msg, CHANNEL) + + assert counter.site_timeserials['site1'] == '01' + + +# UTS: objects/unit/RTLC7c/local-source-no-serial-update-0 +def test_rtlc7c_local_source_no_serial_update(): + counter = InternalLiveCounter(COUNTER_ID) + + msg = object_message(build_counter_inc(COUNTER_ID, 5, '01', 'site1')) + counter.apply_operation(msg, LOCAL) + + assert counter.site_timeserials == {} + assert counter.data == 5 + + +# UTS: objects/unit/RTLC7g/apply-returns-true-0 +def test_rtlc7g_apply_returns_true(): + counter = InternalLiveCounter(COUNTER_ID) + + msg = object_message(build_counter_inc(COUNTER_ID, 5, '01', 'site1')) + result = counter.apply_operation(msg, CHANNEL) + + assert result is True + + +# UTS: objects/unit/RTLO5/object-delete-tombstones-0 +def test_rtlo5_object_delete_tombstones(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 42 + counter.site_timeserials = {'site1': '00'} + updates = capture_updates(counter) + + msg = object_message(build_object_delete(COUNTER_ID, '01', 'site1', 1700000000000)) + applied = counter.apply_operation(msg, CHANNEL) + + assert counter.is_tombstone is True + assert counter.data == 0 + assert counter.tombstoned_at == 1700000000000 + update = _emitted_update(applied, updates) + assert update.update.amount == -42 + assert update.tombstone is True + assert update.object_message is msg + + +# UTS: objects/unit/RTLO5/tombstone-zero-value-counter-emits-update-0 +def test_rtlo5_tombstone_zero_value_counter_emits_update(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 0 + counter.site_timeserials = {'site1': '00'} + updates = capture_updates(counter) + + msg = object_message(build_object_delete(COUNTER_ID, '01', 'site1', 1700000000000)) + applied = counter.apply_operation(msg, CHANNEL) + + assert counter.is_tombstone is True + assert counter.data == 0 + update = _emitted_update(applied, updates) + # The RTLC14c zero-delta exception does not apply to a tombstone diff (RTLO4e5) + assert update.noop is False + assert update.tombstone is True + assert update.update.amount == 0 + assert update.object_message is msg + + +# UTS: objects/unit/RTLC7e/tombstoned-reject-ops-0 +def test_rtlc7e_tombstoned_reject_ops(): + counter = InternalLiveCounter(COUNTER_ID) + counter.is_tombstone = True + counter.tombstoned_at = 1700000000000 + updates = capture_updates(counter) + + msg = object_message(build_counter_inc(COUNTER_ID, 5, '01', 'site1')) + result = counter.apply_operation(msg, CHANNEL) + + _assert_not_applied(result, updates) + assert counter.data == 0 + + +# UTS: objects/unit/RTLO6/tombstoned-at-from-serial-timestamp-0 +def test_rtlo6_tombstoned_at_from_serial_timestamp(): + counter = InternalLiveCounter(COUNTER_ID) + + msg = object_message(build_object_delete(COUNTER_ID, '01', 'site1', 1700000050000)) + counter.apply_operation(msg, CHANNEL) + + assert counter.tombstoned_at == 1700000050000 + + +# UTS: objects/unit/RTLO6/tombstoned-at-local-clock-0 +def test_rtlo6_tombstoned_at_local_clock(): + # The local clock is the counter's own. A fake one, at an epoch no fixture uses, holds + # still between the two readings, so the bounds pin the value RTLO6b must take. + clock = FakeClock(epoch_ms=1_700_000_123_456) + counter = InternalLiveCounter(COUNTER_ID, clock=clock) + before_time = clock.now_ms() + + msg = object_message(build_object_delete(COUNTER_ID, '01', 'site1')) + counter.apply_operation(msg, CHANNEL) + + after_time = clock.now_ms() + assert counter.tombstoned_at >= before_time + assert counter.tombstoned_at <= after_time + + +# UTS: objects/unit/RTLC7d3/unsupported-action-0 +def test_rtlc7d3_unsupported_action(): + counter = InternalLiveCounter(COUNTER_ID) + updates = capture_updates(counter) + + msg = object_message(build_map_set(COUNTER_ID, 'x', {'string': 'y'}, '01', 'site1')) + result = counter.apply_operation(msg, CHANNEL) + + _assert_not_applied(result, updates) + assert counter.data == 0 + + +# UTS: objects/unit/RTLC6/replace-data-basic-0 +def test_rtlc6_replace_data_basic(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 10 + counter.create_operation_is_merged = True + counter.site_timeserials = {'site1': '00'} + + state_msg = object_message(build_object_state(COUNTER_ID, {'site2': '05'}, counter={'count': 50})) + update = counter.replace_data(state_msg) + + assert counter.data == 50 + assert counter.site_timeserials == {'site2': '05'} + assert counter.create_operation_is_merged is False + assert update.update.amount == 40 + assert update.object_message is state_msg + + +# UTS: objects/unit/RTLC6/replace-data-with-create-op-0 +def test_rtlc6_replace_data_with_create_op(): + counter = InternalLiveCounter(COUNTER_ID) + + state_msg = object_message(build_object_state( + COUNTER_ID, {'site1': '01'}, counter={'count': 100}, create_op={'counterCreate': {'count': 50}})) + update = counter.replace_data(state_msg) + + assert counter.data == 150 + assert counter.create_operation_is_merged is True + assert update.update.amount == 150 + assert update.object_message is state_msg + + +# UTS: objects/unit/RTLC6e/replace-data-tombstoned-noop-0 +def test_rtlc6e_replace_data_tombstoned_noop(): + counter = InternalLiveCounter(COUNTER_ID) + counter.is_tombstone = True + counter.tombstoned_at = 1700000000000 + counter.data = 0 + + state_msg = object_message(build_object_state(COUNTER_ID, {'site1': '01'}, counter={'count': 999})) + update = counter.replace_data(state_msg) + + assert counter.data == 0 + assert update.noop is True + + +# UTS: objects/unit/RTLC6f/replace-data-tombstone-flag-0 +def test_rtlc6f_replace_data_tombstone_flag(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 30 + + state_msg = object_message(build_object_state( + COUNTER_ID, {'site1': '01'}, counter={'count': 0}, tombstone=True)) + update = counter.replace_data(state_msg) + + assert counter.is_tombstone is True + assert counter.data == 0 + assert update.update.amount == -30 + assert update.tombstone is True + assert update.object_message is state_msg + + +# UTS: objects/unit/RTLC6/replace-data-missing-count-0 +def test_rtlc6_replace_data_missing_count(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 42 + + state_msg = object_message(build_object_state(COUNTER_ID, {'site1': '01'}, counter={})) + update = counter.replace_data(state_msg) + + assert counter.data == 0 + assert update.update.amount == -42 + assert update.object_message is state_msg + + +# UTS: objects/unit/RTLC14/diff-calculation-0 +def test_rtlc14_diff_calculation(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 20 + + state_msg = object_message(build_object_state(COUNTER_ID, {'site1': '01'}, counter={'count': 75})) + update = counter.replace_data(state_msg) + + assert update.update.amount == 55 + assert update.object_message is state_msg + + +# UTS: objects/unit/RTLC14c/zero-delta-diff-is-noop-0 +def test_rtlc14c_zero_delta_diff_is_noop(): + counter = InternalLiveCounter(COUNTER_ID) + counter.data = 100 + + state_msg = object_message(build_object_state(COUNTER_ID, {'site1': '01'}, counter={'count': 100})) + update = counter.replace_data(state_msg) + + assert update.noop is True + assert counter.data == 100 + + +# UTS: objects/unit/RTLC8/create-then-inc-0 +def test_rtlc8_create_then_inc(): + counter = InternalLiveCounter(COUNTER_ID) + + counter.apply_operation( + object_message(build_counter_create(COUNTER_ID, {'count': 100}, '01', 'site1')), CHANNEL) + counter.apply_operation(object_message(build_counter_inc(COUNTER_ID, 25, '02', 'site1')), CHANNEL) + + assert counter.data == 125 + assert counter.create_operation_is_merged is True + + +# UTS: objects/unit/RTLO3/live-object-init-properties-0 +def test_rtlo3_live_object_init_properties(): + counter = InternalLiveCounter('counter:test@2000') + + assert counter.object_id == 'counter:test@2000' + assert counter.site_timeserials == {} + assert counter.create_operation_is_merged is False + assert counter.is_tombstone is False + assert counter.tombstoned_at is None diff --git a/test/uts/objects/unit/internal_live_map_api_test.py b/test/uts/objects/unit/internal_live_map_api_test.py new file mode 100644 index 00000000..e5dc7f9a --- /dev/null +++ b/test/uts/objects/unit/internal_live_map_api_test.py @@ -0,0 +1,300 @@ +"""Derived from uts/objects/unit/internal_live_map_api.md in ably/specification. + +Spec points: RTLM5, RTLM5d2, RTLM10, RTLM10d, RTLM11, RTLM11d, RTLM12, RTLM20, RTLM20a3, +RTLM20e1, RTLM20e2, RTLM20e3, RTLM20e6, RTLM20e7b, RTLM20e7c, RTLM20e7d, RTLM20e7e, +RTLM20e7f, RTLM20e7g, RTLM20e7g1, RTLM20e7g2, RTLM20h1, RTLM20h2, RTLM21, RTLM21e1, +RTLM21e2, RTLM21e5, RTLMV4c, RTLMV4d1, RTLMV4d2, RTLCV4 + +The specification reaches `InternalLiveMap` through the untyped `PathObject`; ably-python +partitions it by type (RTTS). `root` is already a `LiveMapPathObject`, so `size`, +`entries`, `keys`, `set` and `remove` are called on it directly, and a primitive or counter +below it is read through `as_primitive()` or `as_live_counter()`. + +The tests that capture what the client sends use the standard synced-channel mock, which +is the specification's hand-written one: it records each OBJECT message and then ACKs it +with `ack_serial(msgSerial, i)`, so every awaited write resolves (RTO20). Captured messages +are the decoded JSON wire, where a `json` value is a JSON-encoded string (OD2g), a `bytes` +value is base64, and an action is its number. Each captured operation is also checked +against the whole protocol v6 shape: a primitive MAP_SET or a MAP_REMOVE is compared +exactly, and a create carries only its `*CreateWithObjectId` payload, the `CounterCreate` +or `MapCreate` it was derived from staying local (RTLCV4g5, RTLMV4j5). +""" + +import json + +import pytest + +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction +from ably.pubsub.liveobjects.valuetypes import LiveCounter, LiveMap +from ably.pubsub.util.exceptions import AblyException +from test.uts.objects.helpers.standard_test_pool import ( + LWW, + objects_connected_message, + setup_synced_channel, + standard_mock_websocket, +) + + +def _capturing_mock_websocket(captured): + """The specification's capturing mock: the standard synced-channel conversation, with each + OBJECT message the client sends appended to `captured` before it is ACKed.""" + return standard_mock_websocket( + on_object=captured.append, + connected=objects_connected_message(connection_key='key-1'), + ) + + +def _map_set_operation(object_id, key, value): + """The v6 wire form of a MAP_SET operation; `value` is the `ObjectData` wire dictionary.""" + return { + 'action': int(ObjectOperationAction.MAP_SET), + 'objectId': object_id, + 'mapSet': {'key': key, 'value': value}, + } + + +def _assert_create_wire_shape(operation, payload_name): + """Asserts that a captured create carries only its v6 `*CreateWithObjectId` payload, with + an `initialValue` string and a nonce, and returns the decoded initial value.""" + assert set(operation) == {'action', 'objectId', payload_name} + payload = operation[payload_name] + assert set(payload) == {'initialValue', 'nonce'} + assert isinstance(payload['nonce'], str) + assert isinstance(payload['initialValue'], str) + return json.loads(payload['initialValue']) + + +# UTS: objects/unit/RTLM5/get-string-value-0 +async def test_rtlm5_get_string_value(): + ctx = await setup_synced_channel('test') + + assert ctx.root.get('name').as_primitive().value() == 'Alice' + assert ctx.root.get('age').as_primitive().value() == 30 + assert ctx.root.get('active').as_primitive().value() is True + + +# UTS: objects/unit/RTLM5/get-nonexistent-key-0 +async def test_rtlm5_get_nonexistent_key(): + ctx = await setup_synced_channel('test') + + # The specification's untyped `value()`: a path that resolves to nothing answers None + # through every typed view (RTTS5d1) + assert ctx.root.get('nonexistent').as_primitive().value() is None + assert ctx.root.get('nonexistent').as_live_counter().value() is None + + +# UTS: objects/unit/RTLM5/get-objectid-reference-0 +async def test_rtlm5_get_objectid_reference(): + ctx = await setup_synced_channel('test') + + assert ctx.root.get('score').as_live_counter().value() == 100 + assert ctx.root.get('profile').get('email').as_primitive().value() == 'alice@example.com' + + +# UTS: objects/unit/RTLM10/size-non-tombstoned-0 +async def test_rtlm10_size_non_tombstoned(): + ctx = await setup_synced_channel('test') + + assert ctx.root.size() == 7 + + +# UTS: objects/unit/RTLM11/entries-yields-pairs-0 +async def test_rtlm11_entries_yields_pairs(): + ctx = await setup_synced_channel('test') + + entries = [] + for key, _path_object in ctx.root.entries(): + entries.append(key) + + assert 'name' in entries + assert 'age' in entries + assert 'active' in entries + assert 'score' in entries + assert 'profile' in entries + assert 'data' in entries + assert 'avatar' in entries + assert len(entries) == 7 + + +# UTS: objects/unit/RTLM12/keys-0 +async def test_rtlm12_keys(): + ctx = await setup_synced_channel('test') + + keys = list(ctx.root.keys()) + + assert len(keys) == 7 + assert 'name' in keys + + +# UTS: objects/unit/RTLM20/set-sends-map-set-0 +async def test_rtlm20_set_sends_map_set(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.set('name', 'Bob') + + assert len(captured) == 1 + obj_msg = captured[0]['state'][0] + assert obj_msg['operation']['action'] == ObjectOperationAction.MAP_SET + assert obj_msg['operation']['objectId'] == 'root' + assert obj_msg['operation']['mapSet']['key'] == 'name' + assert obj_msg['operation']['mapSet']['value']['string'] == 'Bob' + assert obj_msg['operation'] == _map_set_operation('root', 'name', {'string': 'Bob'}) + + +# UTS: objects/unit/RTLM20/set-value-types-0 +async def test_rtlm20_set_value_types(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.set('num_key', 42) + await ctx.root.set('bool_key', False) + await ctx.root.set('json_key', {'nested': True}) + + assert captured[0]['state'][0]['operation']['mapSet']['value']['number'] == 42 + assert captured[1]['state'][0]['operation']['mapSet']['value']['boolean'] is False + # A `json` value travels as a JSON-encoded string (OD2g); the specification compares the + # decoded value + json_value = captured[2]['state'][0]['operation']['mapSet']['value'] + assert json.loads(json_value['json']) == {'nested': True} + + assert captured[0]['state'][0]['operation'] == _map_set_operation('root', 'num_key', {'number': 42}) + assert captured[1]['state'][0]['operation'] == _map_set_operation('root', 'bool_key', {'boolean': False}) + assert set(json_value) == {'json'} + assert isinstance(json_value['json'], str) + + +# UTS: objects/unit/RTLM20e7g/set-counter-value-type-0 +async def test_rtlm20e7g_set_counter_value_type(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.set('new_counter', LiveCounter.create(50)) + + assert len(captured) == 1 + state = captured[0]['state'] + assert len(state) == 2 + assert state[0]['operation']['action'] == ObjectOperationAction.COUNTER_CREATE + assert state[0]['operation']['objectId'].startswith('counter:') + assert state[1]['operation']['action'] == ObjectOperationAction.MAP_SET + assert state[1]['operation']['mapSet']['value']['objectId'] == state[0]['operation']['objectId'] + + initial_value = _assert_create_wire_shape(state[0]['operation'], 'counterCreateWithObjectId') + assert initial_value == {'count': 50} + assert state[1]['operation'] == _map_set_operation( + 'root', 'new_counter', {'objectId': state[0]['operation']['objectId']}) + + +# UTS: objects/unit/RTLM20e7g/set-map-value-type-0 +async def test_rtlm20e7g_set_map_value_type(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.set('nested_map', LiveMap.create({'key1': 'value1'})) + + assert len(captured) == 1 + state = captured[0]['state'] + assert len(state) == 2 + assert state[0]['operation']['action'] == ObjectOperationAction.MAP_CREATE + assert state[0]['operation']['objectId'].startswith('map:') + assert state[1]['operation']['action'] == ObjectOperationAction.MAP_SET + assert state[1]['operation']['mapSet']['key'] == 'nested_map' + assert state[1]['operation']['mapSet']['value']['objectId'] == state[0]['operation']['objectId'] + + initial_value = _assert_create_wire_shape(state[0]['operation'], 'mapCreateWithObjectId') + assert initial_value['semantics'] == LWW + assert set(initial_value['entries']) == {'key1'} + assert initial_value['entries']['key1']['data'] == {'string': 'value1'} + assert state[1]['operation'] == _map_set_operation( + 'root', 'nested_map', {'objectId': state[0]['operation']['objectId']}) + + +# UTS: objects/unit/RTLM20h1/set-nested-value-types-0 +async def test_rtlm20h1_set_nested_value_types(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.set('stats', LiveMap.create({ + 'count': LiveCounter.create(0), + 'label': 'test', + })) + + assert len(captured) == 1 + state = captured[0]['state'] + # COUNTER_CREATE, MAP_CREATE, MAP_SET: depth-first, then the MAP_SET at root + assert len(state) == 3 + assert state[0]['operation']['action'] == ObjectOperationAction.COUNTER_CREATE + assert state[0]['operation']['objectId'].startswith('counter:') + assert state[1]['operation']['action'] == ObjectOperationAction.MAP_CREATE + assert state[1]['operation']['objectId'].startswith('map:') + assert state[2]['operation']['action'] == ObjectOperationAction.MAP_SET + assert state[2]['operation']['mapSet']['key'] == 'stats' + assert state[2]['operation']['mapSet']['value']['objectId'] == state[1]['operation']['objectId'] + + counter_initial_value = _assert_create_wire_shape(state[0]['operation'], 'counterCreateWithObjectId') + assert counter_initial_value == {'count': 0} + # RTLMV4d1: the nested counter's entry references the counter the first message creates + map_initial_value = _assert_create_wire_shape(state[1]['operation'], 'mapCreateWithObjectId') + assert map_initial_value['entries']['count']['data'] == {'objectId': state[0]['operation']['objectId']} + assert map_initial_value['entries']['label']['data'] == {'string': 'test'} + assert state[2]['operation'] == _map_set_operation( + 'root', 'stats', {'objectId': state[1]['operation']['objectId']}) + + +# UTS: objects/unit/RTLM21/remove-sends-map-remove-0 +async def test_rtlm21_remove_sends_map_remove(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.remove('name') + + obj_msg = captured[0]['state'][0] + assert obj_msg['operation']['action'] == ObjectOperationAction.MAP_REMOVE + assert obj_msg['operation']['objectId'] == 'root' + assert obj_msg['operation']['mapRemove']['key'] == 'name' + assert obj_msg['operation'] == { + 'action': int(ObjectOperationAction.MAP_REMOVE), + 'objectId': 'root', + 'mapRemove': {'key': 'name'}, + } + + +# UTS: objects/unit/RTLM20/set-applies-locally-0 +async def test_rtlm20_set_applies_locally(): + ctx = await setup_synced_channel('test') + + await ctx.root.set('name', 'Bob') + + assert ctx.root.get('name').as_primitive().value() == 'Bob' + + +# The specification's rows are JavaScript values. Their Python counterparts: a function is a +# lambda, `undefined` is `None`, and a symbol is an arbitrary `object()`. None of them is a +# value a map entry can hold (RTLM20a3). +INVALID_VALUES = [ + pytest.param(lambda: None, id='function'), + pytest.param(None, id='undefined'), + pytest.param(object(), id='symbol'), +] + + +# UTS: objects/unit/RTLM20/set-invalid-values-table-0 +@pytest.mark.parametrize('value', INVALID_VALUES) +async def test_rtlm20_set_invalid_values_table(value): + ctx = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + await ctx.root.set('key', value) + + assert excinfo.value.code == 40013 + + +# UTS: objects/unit/RTLM20/set-bytes-value-0 +async def test_rtlm20_set_bytes_value(): + captured = [] + ctx = await setup_synced_channel('test', mock_ws=_capturing_mock_websocket(captured)) + + await ctx.root.set('binary_data', bytes([1, 2, 3])) + + assert captured[0]['state'][0]['operation']['mapSet']['value']['bytes'] == 'AQID' + assert captured[0]['state'][0]['operation'] == _map_set_operation('root', 'binary_data', {'bytes': 'AQID'}) diff --git a/test/uts/objects/unit/internal_live_map_test.py b/test/uts/objects/unit/internal_live_map_test.py new file mode 100644 index 00000000..a38d1b90 --- /dev/null +++ b/test/uts/objects/unit/internal_live_map_test.py @@ -0,0 +1,928 @@ +"""Derived from uts/objects/unit/internal_live_map.md in ably/specification. + +Spec points: RTLM1-RTLM9, RTLM14-RTLM16, RTLM18, RTLM19, RTLM22-RTLM25, RTLO3, RTLO4a, RTLO4e, +RTLO4g, RTLO4h, RTLO5, RTLO6 + +The specification drives an `InternalLiveMap` directly and reads the update from +`update = map.applyOperation(...)`, or `false` when the operation is rejected. +`apply_operation` returns whether the operation applied (RTLM15g) and emits the update +through `notify_updated` (RTLM15d1a and its siblings), so each test records the emitted +updates with `capture_updates` (shape deviation S-1): where the specification reads +`update`, the operation applied and emitted exactly one update, a no-op included; where +it reads `result == false` or `update == false`, the operation did not apply and emitted +nothing. `replace_data` returns its update (RTLM6h), and `diff` is the static +`InternalLiveMap.diff`, as the specification has them. + +Where the specification passes a pool, the map under test reads it but is not the pool's +own root: `ObjectsPool()` creates that (RTO3b1), and the specification builds a second map +with the id `root` alongside it. +""" + +from ably.pubsub.liveobjects.enums import ObjectsOperationSource +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.livemap import InternalLiveMap +from ably.pubsub.liveobjects.objectmessage import ObjectData, ObjectsMapEntry, ObjectsMapSemantics +from ably.pubsub.liveobjects.objectspool import ObjectsPool +from test.uts.objects.helpers.standard_test_pool import ( + LWW, + build_counter_inc, + build_map_clear, + build_map_create, + build_map_remove, + build_map_set, + build_object_delete, + build_object_state, + capture_updates, + object_message, +) + +CHANNEL = ObjectsOperationSource.CHANNEL + + +def _emitted_update(applied, updates): + """S-1: the specification's `update = map.applyOperation(...)`. + + An operation that applies returns True and emits exactly one update, a no-op included. + """ + assert applied is True, 'expected the operation to apply' + assert len(updates) == 1, f'expected the operation to emit exactly one update, got {updates!r}' + return updates[0] + + +def _assert_not_applied(applied, updates): + """S-1: the specification's `result == false`: the operation did not apply and emitted nothing.""" + assert applied is False + assert updates == [] + + +def _references(child, parent_id, key): + """Whether `child.parent_references` records the map `parent_id` referencing it at `key`. + + `not _references(...)` is the specification's + `parent_id NOT IN child.parentReferences OR key NOT IN child.parentReferences[parent_id]`. + """ + return key in child.parent_references.get(parent_id, set()) + + +# UTS: objects/unit/RTLM4/zero-value-0 +def test_rtlm4_zero_value(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + + assert live_map.data == {} + assert live_map.clear_timeserial is None + assert live_map.is_tombstone is False + assert live_map.create_operation_is_merged is False + assert live_map.site_timeserials == {} + + +# UTS: objects/unit/RTLM7/map-set-new-entry-0 +def test_rtlm7_map_set_new_entry(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'name', {'string': 'Alice'}, '01', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Alice') + assert live_map.data['name'].timeserial == '01' + assert live_map.data['name'].tombstone is False + update = _emitted_update(applied, updates) + assert update.update == {'name': 'updated'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM7/map-set-update-entry-0 +def test_rtlm7_map_set_update_entry(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'name', {'string': 'Bob'}, '02', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Bob') + assert live_map.data['name'].timeserial == '02' + update = _emitted_update(applied, updates) + assert update.update == {'name': 'updated'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM9/lww-reject-stale-0 +def test_rtlm9_lww_reject_stale(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='05', tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'name', {'string': 'Bob'}, '03', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Alice') + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLM9/lww-reject-equal-0 +def test_rtlm9_lww_reject_equal(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='05', tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'name', {'string': 'Bob'}, '05', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Alice') + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLM9b/both-empty-reject-0 +def test_rtlm9b_both_empty_reject(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='', tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'name', {'string': 'Bob'}, '', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Alice') + # The empty ObjectMessage.serial fails the object-level gate (RTLO4a3) before the + # entry-level RTLM9b comparison is reached, so apply_operation returns False (RTLM15b). + _assert_not_applied(applied, updates) + # RTLM9b itself, which apply_operation cannot reach: an empty or missing entry serial + # and operation serial are equal, so the operation is not applied + assert InternalLiveMap.can_apply_map_operation('', '') is False + assert InternalLiveMap.can_apply_map_operation(None, None) is False + + +# UTS: objects/unit/RTLM9d/missing-entry-serial-allows-0 +def test_rtlm9d_missing_entry_serial_allows(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial=None, tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'name', {'string': 'Bob'}, '01', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Bob') + update = _emitted_update(applied, updates) + assert update.update == {'name': 'updated'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM7h/map-set-clear-timeserial-floor-0 +def test_rtlm7h_map_set_clear_timeserial_floor(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.clear_timeserial = '05' + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'name', {'string': 'Alice'}, '03', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert 'name' not in live_map.data + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLM7g/map-set-objectid-creates-zero-value-0 +def test_rtlm7g_map_set_objectid_creates_zero_value(): + pool = ObjectsPool() + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + + msg = object_message(build_map_set('root', 'score', {'objectId': 'counter:new@2000'}, '01', 'site1')) + live_map.apply_operation(msg, CHANNEL) + + assert 'counter:new@2000' in pool + assert isinstance(pool['counter:new@2000'], InternalLiveCounter) + assert pool['counter:new@2000'].data == 0 + + +# UTS: objects/unit/RTLM8/map-remove-existing-0 +def test_rtlm8_map_remove_existing(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_remove('root', 'name', '02', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data is None + assert live_map.data['name'].tombstone is True + assert live_map.data['name'].timeserial == '02' + assert live_map.data['name'].tombstoned_at == 1700000000000 + update = _emitted_update(applied, updates) + assert update.update == {'name': 'removed'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM8/map-remove-nonexistent-0 +def test_rtlm8_map_remove_nonexistent(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + updates = capture_updates(live_map) + + msg = object_message(build_map_remove('root', 'ghost', '01', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['ghost'].tombstone is True + assert live_map.data['ghost'].tombstoned_at == 1700000000000 + update = _emitted_update(applied, updates) + assert update.update == {'ghost': 'removed'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM8g/map-remove-clear-timeserial-floor-0 +def test_rtlm8g_map_remove_clear_timeserial_floor(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.clear_timeserial = '05' + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='04', tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_remove('root', 'name', '03', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Alice') + assert live_map.data['name'].tombstone is False + update = _emitted_update(applied, updates) + assert update.noop is True + + # UTS SPEC ERROR: the entry's own timeserial ('04') is later than the operation's ('03'), + # so LWW (RTLM8a1, RTLM9e) rejects the remove with or without the clear floor, and the + # steps above pass either way. A remove for a key with no entry, still at or below the + # floor, reaches RTLM8g alone: without it the key would gain a tombstoned entry (RTLM8b). + ghost_updates = capture_updates(live_map) + ghost_msg = object_message(build_map_remove('root', 'ghost', '04', 'site1', 1700000000000)) + ghost_applied = live_map.apply_operation(ghost_msg, CHANNEL) + + assert 'ghost' not in live_map.data + ghost_update = _emitted_update(ghost_applied, ghost_updates) + assert ghost_update.noop is True + + +# UTS: objects/unit/RTLM24/map-clear-basic-0 +def test_rtlm24_map_clear_basic(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'old': ObjectsMapEntry(data=ObjectData(string='old'), timeserial='02', tombstone=False), + 'new': ObjectsMapEntry(data=ObjectData(string='new'), timeserial='06', tombstone=False), + 'same': ObjectsMapEntry(data=ObjectData(string='same'), timeserial='04', tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_clear('root', '04', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.clear_timeserial == '04' + assert 'old' not in live_map.data + # RTLM24e1: an entry is removed only if the clear serial is lexicographically greater + # than the entry's timeserial; 'same' has the clear serial itself, so it is kept + assert 'same' in live_map.data + assert 'new' in live_map.data + update = _emitted_update(applied, updates) + assert update.update == {'old': 'removed'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM24c/map-clear-stale-0 +def test_rtlm24c_map_clear_stale(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.clear_timeserial = '10' + updates = capture_updates(live_map) + + msg = object_message(build_map_clear('root', '05', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.clear_timeserial == '10' + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLM16/map-create-merge-0 +def test_rtlm16_map_create_merge(): + live_map = InternalLiveMap('map:test@1000', ObjectsMapSemantics.LWW) + updates = capture_updates(live_map) + + msg = object_message(build_map_create('map:test@1000', { + 'semantics': LWW, + 'entries': { + 'name': {'data': {'string': 'Alice'}, 'timeserial': '01'}, + 'removed_key': {'tombstone': True, 'timeserial': '01', 'serialTimestamp': 1700000000000}, + }, + }, '02', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Alice') + assert live_map.data['removed_key'].tombstone is True + assert live_map.create_operation_is_merged is True + update = _emitted_update(applied, updates) + assert update.update == {'name': 'updated', 'removed_key': 'removed'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM16b/map-create-already-merged-0 +def test_rtlm16b_map_create_already_merged(): + live_map = InternalLiveMap('map:test@1000', ObjectsMapSemantics.LWW) + live_map.create_operation_is_merged = True + live_map.site_timeserials = {'site1': '00'} + updates = capture_updates(live_map) + + msg = object_message(build_map_create('map:test@1000', { + 'semantics': LWW, + 'entries': {'name': {'data': {'string': 'Bob'}, 'timeserial': '01'}}, + }, '01', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert 'name' not in live_map.data + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLM15c/channel-source-updates-serials-0 +def test_rtlm15c_channel_source_updates_serials(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + + msg = object_message(build_map_set('root', 'x', {'number': 1}, '01', 'site1')) + live_map.apply_operation(msg, CHANNEL) + + assert live_map.site_timeserials['site1'] == '01' + + +# UTS: objects/unit/RTLM15e/tombstoned-reject-ops-0 +def test_rtlm15e_tombstoned_reject_ops(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.is_tombstone = True + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'x', {'number': 1}, '01', 'site1')) + result = live_map.apply_operation(msg, CHANNEL) + + _assert_not_applied(result, updates) + assert live_map.data == {} + + +# UTS: objects/unit/RTLO5/object-delete-tombstones-map-0 +def test_rtlo5_object_delete_tombstones_map(): + live_map = InternalLiveMap('map:test@1000', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + 'age': ObjectsMapEntry(data=ObjectData(number=30), timeserial='01', tombstone=False), + } + live_map.site_timeserials = {'site1': '00'} + updates = capture_updates(live_map) + + msg = object_message(build_object_delete('map:test@1000', '01', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.is_tombstone is True + assert live_map.data == {} + update = _emitted_update(applied, updates) + assert update.update == {'name': 'removed', 'age': 'removed'} + assert update.tombstone is True + assert update.object_message is msg + + +# UTS: objects/unit/RTLO5/tombstone-empty-map-emits-update-0 +def test_rtlo5_tombstone_empty_map_emits_update(): + live_map = InternalLiveMap('map:test@1000', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=True, + tombstoned_at=1600000000000), + 'age': ObjectsMapEntry(data=ObjectData(number=30), timeserial='01', tombstone=True, + tombstoned_at=1600000000000), + } + live_map.site_timeserials = {'site1': '00'} + updates = capture_updates(live_map) + + msg = object_message(build_object_delete('map:test@1000', '01', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.is_tombstone is True + assert live_map.data == {} + update = _emitted_update(applied, updates) + # The RTLM22c empty-diff exception does not apply to a tombstone diff (RTLO4e5) + assert update.noop is False + assert update.tombstone is True + assert update.update == {} + assert update.object_message is msg + + +# UTS: objects/unit/RTLO4e10/object-delete-root-noop-0 +def test_rtlo4e10_object_delete_root_noop(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + live_map.site_timeserials = {'site1': '00'} + updates = capture_updates(live_map) + + msg = object_message(build_object_delete('root', '01', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.is_tombstone is False + # The data is untouched + assert live_map.data['name'].data.string == 'Alice' + update = _emitted_update(applied, updates) + assert update.noop is True + + +# UTS: objects/unit/RTLM14/tombstone-check-objectid-ref-0 +def test_rtlm14_tombstone_check_objectid_ref(): + pool = ObjectsPool() + tombstoned_counter = InternalLiveCounter('counter:dead@1000') + tombstoned_counter.is_tombstone = True + pool['counter:dead@1000'] = tombstoned_counter + + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'alive': ObjectsMapEntry(data=ObjectData(string='ok'), timeserial='01', tombstone=False), + 'dead_entry': ObjectsMapEntry(data=None, timeserial='01', tombstone=True), + 'dead_ref': ObjectsMapEntry(data=ObjectData(object_id='counter:dead@1000'), timeserial='01', + tombstone=False), + } + + assert live_map.is_entry_tombstoned(live_map.data['alive']) is False + assert live_map.is_entry_tombstoned(live_map.data['dead_entry']) is True + assert live_map.is_entry_tombstoned(live_map.data['dead_ref']) is True + + +# UTS: objects/unit/RTLM6/replace-data-basic-0 +def test_rtlm6_replace_data_basic(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'old': ObjectsMapEntry(data=ObjectData(string='old'), timeserial='01', tombstone=False), + } + live_map.create_operation_is_merged = True + + state_msg = object_message(build_object_state('root', {'site2': '05'}, map={ + 'semantics': LWW, + 'clearTimeserial': '03', + 'entries': { + 'new': {'data': {'string': 'new'}, 'timeserial': '04', 'tombstone': False}, + }, + })) + update = live_map.replace_data(state_msg) + + assert live_map.site_timeserials == {'site2': '05'} + assert live_map.create_operation_is_merged is False + assert live_map.clear_timeserial == '03' + assert 'old' not in live_map.data + assert live_map.data['new'].data == ObjectData(string='new') + assert update.update == {'old': 'removed', 'new': 'updated'} + assert update.object_message is state_msg + + +# UTS: objects/unit/RTLM6c1/replace-data-tombstoned-entries-0 +def test_rtlm6c1_replace_data_tombstoned_entries(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + + state_msg = object_message(build_object_state('root', {'site1': '01'}, map={ + 'semantics': LWW, + 'entries': { + 'dead': {'tombstone': True, 'timeserial': '01', 'serialTimestamp': 1700000050000}, + }, + })) + live_map.replace_data(state_msg) + + assert live_map.data['dead'].tombstoned_at == 1700000050000 + + +# UTS: objects/unit/RTLM6d/replace-data-with-create-op-0 +def test_rtlm6d_replace_data_with_create_op(): + live_map = InternalLiveMap('map:test@1000', ObjectsMapSemantics.LWW) + + state_msg = object_message(build_object_state('map:test@1000', {'site1': '01'}, map={ + 'semantics': LWW, + 'entries': { + 'from_sync': {'data': {'string': 'synced'}, 'timeserial': '01'}, + }, + }, create_op={ + 'mapCreate': { + 'semantics': LWW, + 'entries': { + 'from_create': {'data': {'string': 'created'}, 'timeserial': '00'}, + }, + }, + })) + live_map.replace_data(state_msg) + + assert live_map.data['from_sync'].data == ObjectData(string='synced') + assert live_map.data['from_create'].data == ObjectData(string='created') + assert live_map.create_operation_is_merged is True + + +# UTS: objects/unit/RTLM6f/replace-data-tombstone-flag-0 +def test_rtlm6f_replace_data_tombstone_flag(): + live_map = InternalLiveMap('map:test@1000', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + + state_msg = object_message(build_object_state( + 'map:test@1000', {'site1': '01'}, map={'semantics': LWW, 'entries': {}}, tombstone=True)) + update = live_map.replace_data(state_msg) + + assert live_map.is_tombstone is True + assert live_map.data == {} + assert update.update == {'name': 'removed'} + assert update.tombstone is True + assert update.object_message is state_msg + + +# UTS: objects/unit/RTLO4e10/replace-data-tombstone-root-noop-0 +def test_rtlo4e10_replace_data_tombstone_root_noop(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + + state_msg = object_message(build_object_state( + 'root', {'site1': '01'}, map={'semantics': LWW, 'entries': {}}, tombstone=True)) + update = live_map.replace_data(state_msg) + + assert live_map.is_tombstone is False + # The data is untouched + assert live_map.data['name'].data.string == 'Alice' + assert update.noop is True + + +# UTS: objects/unit/RTLM19/gc-tombstoned-entries-0 +def test_rtlm19_gc_tombstoned_entries(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + grace_period = 86400000 + now = 1700100000000 + + live_map.data = { + 'recent_dead': ObjectsMapEntry(data=None, timeserial='01', tombstone=True, tombstoned_at=now - 1000), + 'old_dead': ObjectsMapEntry(data=None, timeserial='01', tombstone=True, + tombstoned_at=now - grace_period - 1), + 'alive': ObjectsMapEntry(data=ObjectData(string='ok'), timeserial='01', tombstone=False), + } + + live_map.gc_tombstoned_entries(grace_period, now) + + assert 'recent_dead' in live_map.data + assert 'old_dead' not in live_map.data + assert 'alive' in live_map.data + + +# UTS: objects/unit/RTLM22/diff-calculation-0 +def test_rtlm22_diff_calculation(): + previous_data = { + 'removed': ObjectsMapEntry(data=ObjectData(string='gone'), timeserial='01', tombstone=False), + 'changed': ObjectsMapEntry(data=ObjectData(string='old'), timeserial='01', tombstone=False), + 'unchanged': ObjectsMapEntry(data=ObjectData(string='same'), timeserial='01', tombstone=False), + 'was_dead': ObjectsMapEntry(data=None, timeserial='01', tombstone=True), + } + new_data = { + 'added': ObjectsMapEntry(data=ObjectData(string='new'), timeserial='02', tombstone=False), + 'changed': ObjectsMapEntry(data=ObjectData(string='new_val'), timeserial='02', tombstone=False), + 'unchanged': ObjectsMapEntry(data=ObjectData(string='same'), timeserial='01', tombstone=False), + 'now_dead': ObjectsMapEntry(data=None, timeserial='02', tombstone=True), + } + + update = InternalLiveMap.diff(previous_data, new_data) + + assert update.update['removed'] == 'removed' + assert update.update['added'] == 'updated' + assert update.update['changed'] == 'updated' + assert 'unchanged' not in update.update + assert 'was_dead' not in update.update + assert 'now_dead' not in update.update + + +# UTS: objects/unit/RTLM22c/empty-diff-is-noop-0 +def test_rtlm22c_empty_diff_is_noop(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='alice'), timeserial='01', tombstone=False), + } + + state_msg = object_message(build_object_state('root', {'site1': '02'}, map={ + 'semantics': LWW, + 'entries': { + 'name': {'data': {'string': 'alice'}, 'timeserial': '02', 'tombstone': False}, + }, + })) + update = live_map.replace_data(state_msg) + + assert update.noop is True + assert live_map.data['name'].data == ObjectData(string='alice') + + +# UTS: objects/unit/RTLM15d4/unsupported-action-0 +def test_rtlm15d4_unsupported_action(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + updates = capture_updates(live_map) + + msg = object_message(build_counter_inc('root', 5, '01', 'site1')) + result = live_map.apply_operation(msg, CHANNEL) + + _assert_not_applied(result, updates) + + +# UTS: objects/unit/RTLM6i/replace-data-resets-clear-timeserial-0 +def test_rtlm6i_replace_data_resets_clear_timeserial(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.clear_timeserial = '05' + live_map.data = { + 'x': ObjectsMapEntry(data=ObjectData(number=1), timeserial='03', tombstone=False), + } + + state_msg = object_message(build_object_state('root', {'site1': '01'}, map={ + 'semantics': LWW, + 'entries': { + 'y': {'data': {'number': 2}, 'timeserial': '01'}, + }, + })) + live_map.replace_data(state_msg) + + assert live_map.clear_timeserial is None + assert 'y' in live_map.data + + +# UTS: objects/unit/RTLM14c/tombstoned-ref-yields-null-0 +def test_rtlm14c_tombstoned_ref_yields_null(): + pool = ObjectsPool() + tombstoned_counter = InternalLiveCounter('counter:dead@1000') + tombstoned_counter.is_tombstone = True + pool['counter:dead@1000'] = tombstoned_counter + + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'ref': ObjectsMapEntry(data=ObjectData(object_id='counter:dead@1000'), timeserial='01', tombstone=False), + } + + # The entry itself is not tombstoned, but the object it references is + assert live_map.data['ref'].tombstone is False + # RTLM14c makes the entry tombstoned, so size() does not count it + assert live_map.size() == 0 + assert live_map.get('ref') is None + + +# UTS: objects/unit/RTLM7/map-set-revives-tombstoned-0 +def test_rtlm7_map_set_revives_tombstoned(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'name': ObjectsMapEntry(data=None, timeserial='01', tombstone=True, tombstoned_at=1700000000000), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'name', {'string': 'Alice'}, '02', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].data == ObjectData(string='Alice') + assert live_map.data['name'].tombstone is False + assert live_map.data['name'].tombstoned_at is None + update = _emitted_update(applied, updates) + assert update.update == {'name': 'updated'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM24/map-clear-preserves-newer-0 +def test_rtlm24_map_clear_preserves_newer(): + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW) + live_map.data = { + 'before': ObjectsMapEntry(data=ObjectData(string='a'), timeserial='03', tombstone=False), + 'after': ObjectsMapEntry(data=ObjectData(string='b'), timeserial='07', tombstone=False), + 'no_ts': ObjectsMapEntry(data=ObjectData(string='c'), timeserial=None, tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_clear('root', '05', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert 'before' not in live_map.data + assert 'no_ts' not in live_map.data + assert live_map.data['after'].data == ObjectData(string='b') + update = _emitted_update(applied, updates) + assert 'before' in update.update + assert 'no_ts' in update.update + assert 'after' not in update.update + assert update.object_message is msg + + +# UTS: objects/unit/RTLM7a3/map-set-overwrite-objectid-parent-refs-0 +def test_rtlm7a3_map_set_overwrite_objectid_parent_refs(): + pool = ObjectsPool() + old_counter = InternalLiveCounter('counter:old@1000') + new_counter = InternalLiveCounter('counter:new@2000') + pool['counter:old@1000'] = old_counter + pool['counter:new@2000'] = new_counter + + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'ref': ObjectsMapEntry(data=ObjectData(object_id='counter:old@1000'), timeserial='01', tombstone=False), + } + # The existing reference, as the map would have recorded it + old_counter.parent_references = {'root': {'ref'}} + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'ref', {'objectId': 'counter:new@2000'}, '02', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['ref'].data == ObjectData(object_id='counter:new@2000') + # removeParentReference was called on the old child + assert not _references(old_counter, 'root', 'ref') + # addParentReference was called on the new child + assert 'root' in new_counter.parent_references + assert 'ref' in new_counter.parent_references['root'] + update = _emitted_update(applied, updates) + assert update.update == {'ref': 'updated'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM7g2/map-set-new-entry-add-parent-ref-0 +def test_rtlm7g2_map_set_new_entry_add_parent_ref(): + pool = ObjectsPool() + child_counter = InternalLiveCounter('counter:child@1000') + pool['counter:child@1000'] = child_counter + + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'score', {'objectId': 'counter:child@1000'}, '01', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['score'].data == ObjectData(object_id='counter:child@1000') + assert 'root' in child_counter.parent_references + assert 'score' in child_counter.parent_references['root'] + update = _emitted_update(applied, updates) + assert update.object_message is msg + + +# UTS: objects/unit/RTLM7/map-set-primitive-no-parent-refs-0 +def test_rtlm7_map_set_primitive_no_parent_refs(): + pool = ObjectsPool() + old_counter = InternalLiveCounter('counter:old@1000') + pool['counter:old@1000'] = old_counter + + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'ref': ObjectsMapEntry(data=ObjectData(object_id='counter:old@1000'), timeserial='01', tombstone=False), + } + old_counter.parent_references = {'root': {'ref'}} + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'ref', {'string': 'plain_value'}, '02', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['ref'].data == ObjectData(string='plain_value') + # removeParentReference was called on the old child, as the entry referenced it; the new + # value is a primitive, so there is no child to call addParentReference on + assert not _references(old_counter, 'root', 'ref') + update = _emitted_update(applied, updates) + assert update.update == {'ref': 'updated'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM8a3/map-remove-objectid-parent-refs-0 +def test_rtlm8a3_map_remove_objectid_parent_refs(): + pool = ObjectsPool() + child_counter = InternalLiveCounter('counter:child@1000') + pool['counter:child@1000'] = child_counter + + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'score': ObjectsMapEntry(data=ObjectData(object_id='counter:child@1000'), timeserial='01', + tombstone=False), + } + child_counter.parent_references = {'root': {'score'}} + updates = capture_updates(live_map) + + msg = object_message(build_map_remove('root', 'score', '02', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['score'].tombstone is True + # removeParentReference was called on the child + assert not _references(child_counter, 'root', 'score') + update = _emitted_update(applied, updates) + assert update.update == {'score': 'removed'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM8/map-remove-primitive-no-parent-refs-0 +def test_rtlm8_map_remove_primitive_no_parent_refs(): + pool = ObjectsPool() + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + updates = capture_updates(live_map) + + msg = object_message(build_map_remove('root', 'name', '02', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['name'].tombstone is True + update = _emitted_update(applied, updates) + assert update.update == {'name': 'removed'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLM24e1c/map-clear-parent-refs-0 +def test_rtlm24e1c_map_clear_parent_refs(): + pool = ObjectsPool() + counter_a = InternalLiveCounter('counter:a@1000') + counter_b = InternalLiveCounter('counter:b@1000') + pool['counter:a@1000'] = counter_a + pool['counter:b@1000'] = counter_b + + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'ref_a': ObjectsMapEntry(data=ObjectData(object_id='counter:a@1000'), timeserial='02', tombstone=False), + 'ref_b': ObjectsMapEntry(data=ObjectData(object_id='counter:b@1000'), timeserial='02', tombstone=False), + 'primitive': ObjectsMapEntry(data=ObjectData(string='hello'), timeserial='02', tombstone=False), + 'newer': ObjectsMapEntry(data=ObjectData(string='kept'), timeserial='09', tombstone=False), + } + counter_a.parent_references = {'root': {'ref_a'}} + counter_b.parent_references = {'root': {'ref_b'}} + updates = capture_updates(live_map) + + msg = object_message(build_map_clear('root', '05', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + # ref_a, ref_b and primitive are removed (timeserial '02' < '05'); newer is kept ('09' > '05') + assert 'ref_a' not in live_map.data + assert 'ref_b' not in live_map.data + assert 'primitive' not in live_map.data + assert 'newer' in live_map.data + # removeParentReference was called on both child counters + assert not _references(counter_a, 'root', 'ref_a') + assert not _references(counter_b, 'root', 'ref_b') + update = _emitted_update(applied, updates) + assert update.update == {'ref_a': 'removed', 'ref_b': 'removed', 'primitive': 'removed'} + assert update.object_message is msg + + +# UTS: objects/unit/RTLO4e9/tombstone-map-parent-refs-0 +def test_rtlo4e9_tombstone_map_parent_refs(): + pool = ObjectsPool() + child_counter = InternalLiveCounter('counter:child@1000') + child_map = InternalLiveMap('map:child@1000', ObjectsMapSemantics.LWW) + pool['counter:child@1000'] = child_counter + pool['map:child@1000'] = child_map + + live_map = InternalLiveMap('map:test@1000', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'counter_ref': ObjectsMapEntry(data=ObjectData(object_id='counter:child@1000'), timeserial='01', + tombstone=False), + 'map_ref': ObjectsMapEntry(data=ObjectData(object_id='map:child@1000'), timeserial='01', tombstone=False), + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + live_map.site_timeserials = {'site1': '00'} + child_counter.parent_references = {'map:test@1000': {'counter_ref'}} + child_map.parent_references = {'map:test@1000': {'map_ref'}} + updates = capture_updates(live_map) + + msg = object_message(build_object_delete('map:test@1000', '01', 'site1', 1700000000000)) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.is_tombstone is True + assert live_map.data == {} + # removeParentReference was called on both children + assert not _references(child_counter, 'map:test@1000', 'counter_ref') + assert not _references(child_map, 'map:test@1000', 'map_ref') + update = _emitted_update(applied, updates) + assert update.update == {'counter_ref': 'removed', 'map_ref': 'removed', 'name': 'removed'} + assert update.tombstone is True + assert update.object_message is msg + + +# UTS: objects/unit/RTLM7a3/map-set-replace-objectid-both-refs-0 +def test_rtlm7a3_map_set_replace_objectid_both_refs(): + pool = ObjectsPool() + old_map = InternalLiveMap('map:old@1000', ObjectsMapSemantics.LWW) + new_map = InternalLiveMap('map:new@2000', ObjectsMapSemantics.LWW) + pool['map:old@1000'] = old_map + pool['map:new@2000'] = new_map + + live_map = InternalLiveMap('root', ObjectsMapSemantics.LWW, pool=pool) + live_map.data = { + 'child': ObjectsMapEntry(data=ObjectData(object_id='map:old@1000'), timeserial='01', tombstone=False), + } + old_map.parent_references = {'root': {'child'}} + updates = capture_updates(live_map) + + msg = object_message(build_map_set('root', 'child', {'objectId': 'map:new@2000'}, '02', 'site1')) + applied = live_map.apply_operation(msg, CHANNEL) + + assert live_map.data['child'].data == ObjectData(object_id='map:new@2000') + # The old child no longer records the reference from root + assert not _references(old_map, 'root', 'child') + # The new child records it + assert 'root' in new_map.parent_references + assert 'child' in new_map.parent_references['root'] + update = _emitted_update(applied, updates) + assert update.update == {'child': 'updated'} + assert update.object_message is msg diff --git a/test/uts/objects/unit/live_object_subscribe_test.py b/test/uts/objects/unit/live_object_subscribe_test.py new file mode 100644 index 00000000..437bd2b0 --- /dev/null +++ b/test/uts/objects/unit/live_object_subscribe_test.py @@ -0,0 +1,321 @@ +"""Derived from uts/objects/unit/live_object_subscribe.md in ably/specification. + +Spec points: RTLO4b, RTLO4b3, RTLO4b4c1, RTLO4b4c3a, RTLO4b4c3c, RTLO4b4d, RTLO4b4e, +RTLO4b6, RTLO4b7, RTINS16e, RTLC14c + +`LiveObject#subscribe` is exercised through `Instance#subscribe` (RTINS16), as the +specification does. ably-python partitions `Instance` by type (RTTS) and only the live +object views have `subscribe`, so a counter's instance is subscribed through +`as_live_counter()` and the root map's through `as_live_map()`. Each listener receives an +`InstanceSubscriptionEvent`, whose `message` is the public `ObjectMessage`. + +Inbound frames are processed on the transport's read task, so a positive is awaited with +`poll_until`, and an exact count or a "listener did not fire" is asserted only once a +control listener on a later dispatch has fired and the loop has settled (the +specification's negative-assertion quiescence pattern). +""" + +from ably.pubsub.liveobjects import publicmessage +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction +from ably.pubsub.liveobjects.subscription import Subscription +from test.uts.helpers.client import poll_until +from test.uts.helpers.clock import settle +from test.uts.objects.helpers.standard_test_pool import ( + assert_unchanged_after_quiescence, + build_counter_inc, + build_map_set, + build_object_delete, + build_object_message, + build_public_object_message, + remote_serial, + setup_synced_channel, +) + + +# UTS: objects/unit/RTLO4b/subscribe-receives-updates-0 +async def test_rtlo4b_subscribe_receives_updates(): + ctx = await setup_synced_channel('test') + updates = [] + instance = ctx.root.get('score').instance() + sub = instance.as_live_counter().subscribe(lambda event: updates.append(event)) + + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote'), + ])) + await poll_until(lambda: len(updates) >= 1, timeout=5, description='the counter update to be delivered') + await settle() + + assert isinstance(sub, Subscription) + assert len(updates) == 1 + + +# UTS: objects/unit/RTLO4b7/subscribe-returns-subscription-0 +async def test_rtlo4b7_subscribe_returns_subscription(): + ctx = await setup_synced_channel('test') + instance = ctx.root.get('score').instance() + + sub = instance.as_live_counter().subscribe(lambda event: None) + + assert isinstance(sub, Subscription) + assert callable(sub.unsubscribe) + + +# UTS: objects/unit/RTLO4b7/subscription-unsubscribe-stops-delivery-0 +async def test_rtlo4b7_subscription_unsubscribe_stops_delivery(): + ctx = await setup_synced_channel('test') + updates = [] + control = [] + instance = ctx.root.get('score').instance() + sub = instance.as_live_counter().subscribe(lambda event: updates.append(event)) + + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 5, '01', 'remote'), + ])) + await poll_until(lambda: len(updates) >= 1, timeout=5, description='the first update to be delivered') + + sub.unsubscribe() + + # A control listener fires on the same dispatch as the message under test, so once it + # has fired the unsubscribed listener would also have run had it still been registered + instance.as_live_counter().subscribe(lambda event: control.append(event)) + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 10, '02', 'remote'), + ])) + await assert_unchanged_after_quiescence(lambda: len(updates), lambda: len(control) >= 1, + description='the control listener received the second update') + + assert len(updates) == 1 + + +# UTS: objects/unit/RTLO4b7/subscription-unsubscribe-idempotent-0 +async def test_rtlo4b7_subscription_unsubscribe_idempotent(): + ctx = await setup_synced_channel('test') + instance = ctx.root.get('score').instance() + sub = instance.as_live_counter().subscribe(lambda event: None) + + # Neither call raises + sub.unsubscribe() + sub.unsubscribe() + + +# UTS: objects/unit/RTLO4b4c1/noop-no-trigger-0 +async def test_rtlo4b4c1_noop_no_trigger(): + ctx = await setup_synced_channel('test') + updates = [] + control = [] + instance = ctx.root.get('score').instance() + instance.as_live_counter().subscribe(lambda event: updates.append(event)) + + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 5, '01', 'remote'), + ])) + await poll_until(lambda: len(updates) >= 1, timeout=5, description='the first update to be delivered') + + # Serial "02" passes the newness check (RTLO4a6), and an increment with no `number` is + # the no-op (RTLC9h). `number: 0` would be present per RTLC9g and give a real update of 0. + ctx.mock_ws.send_to_client(build_object_message('test', [{ + 'serial': '02', + 'siteCode': 'remote', + 'operation': { + 'action': int(ObjectOperationAction.COUNTER_INC), + 'objectId': 'counter:score@1000', + 'counterInc': {}, + }, + }])) + # The follow-up "03" is dispatched after the no-op, so once its separate control + # listener fires the no-op has certainly been processed + control_sub = instance.as_live_counter().subscribe(lambda event: control.append(event)) + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 3, '03', 'remote'), + ])) + await poll_until(lambda: len(control) >= 1, timeout=5, description='the control listener to receive "03"') + await settle() + control_sub.unsubscribe() + + # The original listener fired for "01" and "03" only; had the no-op fired it would be 3 + assert len(updates) == 2 + + +# UTS: objects/unit/RTLO4b6/subscribe-no-side-effects-0 +async def test_rtlo4b6_subscribe_no_side_effects(): + ctx = await setup_synced_channel('test') + state_before = ctx.channel.state + sync_state_before = ctx.channel.object._sync_state + sent_before = len(ctx.mock_ws.messages_from_client) + instance = ctx.root.get('score').instance() + + instance.as_live_counter().subscribe(lambda event: None) + await settle() + + assert ctx.channel.state == state_before + # Nor does it change the objects' sync state or send anything on the channel + assert ctx.channel.object._sync_state == sync_state_before + assert len(ctx.mock_ws.messages_from_client) == sent_before + + +# UTS: objects/unit/RTLO4b/subscribe-map-update-0 +async def test_rtlo4b_subscribe_map_update(): + ctx = await setup_synced_channel('test') + updates = [] + instance = ctx.root.instance() + instance.as_live_map().subscribe(lambda event: updates.append(event)) + + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(updates) >= 1, timeout=5, description='the map update to be delivered') + await settle() + + assert len(updates) == 1 + + +# UTS: objects/unit/RTLO4b4c3c/tombstone-deregisters-listeners-0 +async def test_rtlo4b4c3c_tombstone_deregisters_listeners(): + ctx = await setup_synced_channel('test') + updates_a = [] + updates_b = [] + control = [] + instance = ctx.root.get('score').instance() + instance.as_live_counter().subscribe(lambda event: updates_a.append(event)) + instance.as_live_counter().subscribe(lambda event: updates_b.append(event)) + + # An OBJECT_DELETE tombstones the counter + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_object_delete('counter:score@1000', '50', 'remote'), + ])) + # Both listeners are awaited on this dispatch before either count is asserted + await poll_until(lambda: len(updates_a) >= 1, timeout=5, description='listener A to receive the tombstone') + await poll_until(lambda: len(updates_b) >= 1, timeout=5, description='listener B to receive the tombstone') + await settle() + + # Both listeners received the tombstone update + assert len(updates_a) == 1 + assert updates_a[0].message.operation.action == ObjectOperationAction.OBJECT_DELETE + assert len(updates_b) == 1 + assert updates_b[0].message.operation.action == ObjectOperationAction.OBJECT_DELETE + + # A tombstoned object ignores further operations (RTLC7e), so no listener on the counter + # can serve as the control. A listener on another live object can: the "52" MAP_SET on + # the profile map is processed after the "51" increment, so once it fires "51" has been too. + control_inst = ctx.root.get('profile').instance() + control_inst.as_live_map().subscribe(lambda event: control.append(event)) + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 3, '51', 'remote'), + ])) + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_map_set('map:profile@1000', 'quiescence_probe', {'string': 'x'}, '52', 'remote'), + ])) + await assert_unchanged_after_quiescence(lambda: (len(updates_a), len(updates_b)), lambda: len(control) >= 1, + description='the control listener on the profile map fired') + + # The tombstone deregistered both listeners (RTLO4b4c3c) + assert len(updates_a) == 1 + assert len(updates_b) == 1 + + +# UTS: objects/unit/RTLO4b4c3c/tombstone-zero-value-counter-tears-down-0 +async def test_rtlo4b4c3c_tombstone_zero_value_counter_tears_down(): + ctx = await setup_synced_channel('test') + updates_a = [] + updates_b = [] + control = [] + instance = ctx.root.get('score').instance() + + # The counter (100 in the standard pool) is driven to 0 before the listeners under test + # are registered, so they observe only the tombstone + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', -100, '40', 'remote'), + ])) + await poll_until(lambda: ctx.root.get('score').as_live_counter().value() == 0, timeout=5, + description='the counter to reach 0') + + instance.as_live_counter().subscribe(lambda event: updates_a.append(event)) + instance.as_live_counter().subscribe(lambda event: updates_b.append(event)) + + # The OBJECT_DELETE tombstones an already-zero counter: a zero-delta diff, which for a + # tombstone is not a no-op (RTLC14c) + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_object_delete('counter:score@1000', '50', 'remote'), + ])) + await poll_until(lambda: len(updates_a) >= 1, timeout=5, description='listener A to receive the tombstone') + await poll_until(lambda: len(updates_b) >= 1, timeout=5, description='listener B to receive the tombstone') + await settle() + + # Both listeners received the tombstone update although the counter's data did not change + assert len(updates_a) == 1 + assert updates_a[0].message.operation.action == ObjectOperationAction.OBJECT_DELETE + assert len(updates_b) == 1 + assert updates_b[0].message.operation.action == ObjectOperationAction.OBJECT_DELETE + + # A separate live object is the quiescence barrier, as in the populated case + control_inst = ctx.root.get('profile').instance() + control_inst.as_live_map().subscribe(lambda event: control.append(event)) + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 3, '51', 'remote'), + ])) + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_map_set('map:profile@1000', 'quiescence_probe', {'string': 'x'}, '52', 'remote'), + ])) + await assert_unchanged_after_quiescence(lambda: (len(updates_a), len(updates_b)), lambda: len(control) >= 1, + description='the control listener on the profile map fired') + + # The tombstone deregistered both listeners (RTLO4b4c3c) + assert len(updates_a) == 1 + assert len(updates_b) == 1 + + +# UTS: objects/unit/RTLO4b4d/update-has-object-message-0 +async def test_rtlo4b4d_update_has_object_message(): + ctx = await setup_synced_channel('test') + updates = [] + instance = ctx.root.get('score').instance() + instance.as_live_counter().subscribe(lambda event: updates.append(event)) + + counter_inc = build_counter_inc('counter:score@1000', 7, '99', 'remote') + ctx.mock_ws.send_to_client(build_object_message('test', [counter_inc])) + await poll_until(lambda: len(updates) >= 1, timeout=5, description='the counter update to be delivered') + await settle() + + assert len(updates) == 1 + assert updates[0].message is not None + assert updates[0].message.serial == '99' + assert updates[0].message.site_code == 'remote' + assert updates[0].message.operation.action == ObjectOperationAction.COUNTER_INC + assert updates[0].message.operation.object_id == 'counter:score@1000' + # RTINS16e2: the message is the public ObjectMessage derived from the source per PAOM3 + assert isinstance(updates[0].message, publicmessage.ObjectMessage) + assert updates[0].message == build_public_object_message(counter_inc, 'test') + + +# UTS: objects/unit/RTLO4b4e/tombstone-flag-true-0 +async def test_rtlo4b4e_tombstone_flag_true(): + ctx = await setup_synced_channel('test') + updates = [] + instance = ctx.root.get('score').instance() + instance.as_live_counter().subscribe(lambda event: updates.append(event)) + + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_object_delete('counter:score@1000', '50', 'remote'), + ])) + await poll_until(lambda: len(updates) >= 1, timeout=5, description='the tombstone update to be delivered') + await settle() + + assert len(updates) == 1 + assert updates[0].message.operation.action == ObjectOperationAction.OBJECT_DELETE + + +# UTS: objects/unit/RTLO4b4e/tombstone-flag-false-0 +async def test_rtlo4b4e_tombstone_flag_false(): + ctx = await setup_synced_channel('test') + updates = [] + instance = ctx.root.get('score').instance() + instance.as_live_counter().subscribe(lambda event: updates.append(event)) + + ctx.mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote'), + ])) + await poll_until(lambda: len(updates) >= 1, timeout=5, description='the counter update to be delivered') + await settle() + + assert len(updates) == 1 + assert updates[0].message.operation.action == ObjectOperationAction.COUNTER_INC diff --git a/test/uts/objects/unit/object_id_test.py b/test/uts/objects/unit/object_id_test.py new file mode 100644 index 00000000..fdbedbeb --- /dev/null +++ b/test/uts/objects/unit/object_id_test.py @@ -0,0 +1,99 @@ +"""Derived from uts/objects/unit/object_id.md in ably/specification. + +Spec points: RTO14, RTO14a1, RTO14b, RTO14b1, RTO14b2, RTO14c + +A pure function: `generate_object_id(object_type, initial_value, nonce, timestamp_ms)` is the +specification's `generateObjectId(type:, initialValue:, nonce:, timestamp:)`, producing +`[type]:[hash]@[timestamp]`. +""" + +import base64 +import hashlib +import re + +from ably.pubsub.liveobjects.objectid import generate_object_id + +# RFC 4648 s.5's URL-safe alphabet, with no padding +BASE64URL_UNPADDED = re.compile(r'[A-Za-z0-9_-]+') + + +def _is_base64url(value): + """Whether `value` is unpadded base64url that decodes: the alphabet, and a length no + encoding produces (one more than a multiple of four) excluded.""" + if not BASE64URL_UNPADDED.fullmatch(value) or len(value) % 4 == 1: + return False + base64.urlsafe_b64decode(value + '=' * (-len(value) % 4)) + return True + + +def _expected_hash(initial_value, nonce): + """RTO14b, computed here independently: the unpadded base64url SHA-256 digest of the + UTF-8 string `initial_value:nonce`.""" + digest = hashlib.sha256(f'{initial_value}:{nonce}'.encode()).digest() + return base64.urlsafe_b64encode(digest).decode('ascii').rstrip('=') + + +# UTS: objects/unit/RTO14/objectid-format-counter-0 +def test_rto14_objectid_format_counter(): + object_id = generate_object_id('counter', '{"counter":{"count":42}}', 'test-nonce-12345678', 1700000000000) + + assert object_id.startswith('counter:') + assert '@1700000000000' in object_id + parts = object_id.split(':') + # The shape is asserted before indexing, so a malformed id fails on an assertion + assert len(parts) == 2 + type_part = parts[0] + rest = parts[1] + hash_and_ts = rest.split('@') + assert len(hash_and_ts) == 2 + hash_part = hash_and_ts[0] + ts_part = hash_and_ts[1] + assert type_part == 'counter' + assert ts_part == '1700000000000' + assert _is_base64url(hash_part) + assert '+' not in hash_part + assert '/' not in hash_part + assert '=' not in hash_part + # The hash itself, which RTO14b1 and RTO14b2 determine and the specification's + # assertions leave open: the digest of the initial value and the nonce, joined by ':' + assert hash_part == _expected_hash('{"counter":{"count":42}}', 'test-nonce-12345678') + + +# UTS: objects/unit/RTO14/objectid-format-map-0 +def test_rto14_objectid_format_map(): + object_id = generate_object_id( + 'map', '{"map":{"semantics":"LWW","entries":{}}}', 'test-nonce-12345678', 1700000000000) + + assert object_id.startswith('map:') + assert '@1700000000000' in object_id + + +# UTS: objects/unit/RTO14/deterministic-0 +def test_rto14_deterministic(): + id1 = generate_object_id('counter', '{"counter":{"count":0}}', 'same-nonce-1234567', 1700000000000) + id2 = generate_object_id('counter', '{"counter":{"count":0}}', 'same-nonce-1234567', 1700000000000) + + assert id1 == id2 + + +# UTS: objects/unit/RTO14/different-nonce-0 +def test_rto14_different_nonce(): + id1 = generate_object_id('counter', '{"counter":{"count":0}}', 'nonce-aaaaaaaaaaaaa', 1700000000000) + id2 = generate_object_id('counter', '{"counter":{"count":0}}', 'nonce-bbbbbbbbbbbbb', 1700000000000) + + assert id1 != id2 + + +# UTS: objects/unit/RTO14b/base64url-encoding-0 +def test_rto14b_base64url_encoding(): + object_id = generate_object_id('counter', '{"counter":{"count":0}}', 'test-nonce-12345678', 1700000000000) + parts = object_id.split(':') + # The shape is asserted before indexing, so a malformed id fails on an assertion + assert len(parts) == 2 + hash_and_ts = parts[1].split('@') + assert len(hash_and_ts) == 2 + hash_part = hash_and_ts[0] + + assert '+' not in hash_part + assert '/' not in hash_part + assert not hash_part.endswith('=') diff --git a/test/uts/objects/unit/objects_pool_test.py b/test/uts/objects/unit/objects_pool_test.py new file mode 100644 index 00000000..ad908356 --- /dev/null +++ b/test/uts/objects/unit/objects_pool_test.py @@ -0,0 +1,687 @@ +"""Derived from uts/objects/unit/objects_pool.md in ably/specification. + +Spec points: RTO3, RTO3a, RTO3b, RTO3b1, RTO4, RTO4a, RTO4b, RTO4b1, RTO4b2, RTO4b2a, RTO4b4, +RTO4c, RTO4d, RTO5, RTO5a1, RTO5a2, RTO5a2a, RTO5a4, RTO5a5, RTO5a6, RTO5c, RTO5c2, RTO5c2a, +RTO5c6, RTO5c7, RTO5c8, RTO5c9, RTO5c10, RTO5c10a, RTO5c10b, RTO5d, RTO5e, RTO5f1, RTO5f2, +RTO5f2a2, RTO5f2b, RTO5f3, RTO6, RTO6a, RTO6b1, RTO6b2, RTO6b3, RTO7, RTO7a, RTO8a, RTO8b, +RTO9a1, RTO9a2a4, RTO9a2b, RTO9a3, RTLM23 + +A pure unit specification: nothing is mocked and nothing is connected. The specification +puts the sync and apply state machine on `ObjectsPool` (`processAttached`, +`processObjectSync`, `processObjectMessage`, `applyObjectMessages`, `syncState`). Here, as +in features.md (RTO4-RTO9, RTO17), it is `RealtimeObject`'s, and the pool is a mapping of +object ids to objects (RTO3a); this is shape deviation S-2. A test that builds only +`pool = ObjectsPool()` and then drives the state machine works on the pool a standalone +`RealtimeObject()` holds (`_driven_pool()` below); one that builds +`RealtimeObject(pool: pool)` does the same with `RealtimeObject(pool=pool)`. The +`_process_*` helpers below are the specification's three `process*` calls. + +The channel hands `RealtimeObject` an ATTACHED's HAS_OBJECTS flag and nothing else (RTO4), +so the `channelSerial` the specification gives its ATTACHED messages goes no further than +the message dictionary. An OBJECT_SYNC's `channelSerial` is the sync cursor (RTO5a1) and is +passed on. + +Updates are emitted synchronously (`LiveObject.notify_updated`), so a listener registered +with `subscribe` has seen every update by the time the call that caused it returns. The +tests that drive a `RealtimeObject` are coroutines only so that an implementation is free to +touch the event loop. +""" + +from ably.pubsub.liveobjects.enums import ObjectsOperationSource, ObjectsSyncState +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.livemap import InternalLiveMap +from ably.pubsub.liveobjects.objectmessage import ObjectData, ObjectOperationAction, ObjectsMapEntry +from ably.pubsub.liveobjects.objectspool import ObjectsPool +from ably.pubsub.liveobjects.realtimeobject import RealtimeObject +from test.uts.objects.helpers.standard_test_pool import ( + ATTACHED, + HAS_OBJECTS, + LWW, + build_counter_inc, + build_map_set, + build_object_message, + build_object_state, + build_object_sync_message, + object_message, + object_messages, + objects_attached_message, +) + + +def _driven_pool(): + """The specification's `pool = ObjectsPool()`, in a test that then drives the sync state machine. + + S-2: the state machine is `RealtimeObject`'s, so the pool is the one a standalone + `RealtimeObject` holds. Returns `(realtime_object, pool)`. + """ + realtime_object = RealtimeObject() + return realtime_object, realtime_object._objects_pool + + +def _process_attached(realtime_object, attached): + """The specification's `pool.processAttached(attached)` (S-2). + + `RealtimeObject` is given the ATTACHED's HAS_OBJECTS flag; a message with no `flags` + has the flag unset (RTO4b). + """ + realtime_object._on_attached(has_objects=bool(attached.get('flags', 0) & HAS_OBJECTS)) + + +def _process_object_sync(realtime_object, protocol_message): + """The specification's `pool.processObjectSync(protocol_message)` (S-2).""" + realtime_object._handle_object_sync_messages( + object_messages(protocol_message), protocol_message.get('channelSerial')) + + +def _process_object_message(realtime_object, protocol_message): + """The specification's `pool.processObjectMessage(protocol_message)` (S-2).""" + realtime_object._handle_object_messages(object_messages(protocol_message)) + + +def _empty_root_state(): + """The root state most syncs carry: an empty LWW map whose create operation adds nothing.""" + return build_object_state('root', {'aaa': 't:0'}, map={'semantics': LWW, 'entries': {}}, + create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}) + + +# UTS: objects/unit/RTO3/pool-init-root-0 +def test_rto3_pool_init_root(): + pool = ObjectsPool() + + assert 'root' in pool + assert isinstance(pool['root'], InternalLiveMap) + assert pool['root'].data == {} + assert pool['root'].object_id == 'root' + + +# UTS: objects/unit/RTO4/attached-has-objects-syncing-0 +async def test_rto4_attached_has_objects_syncing(): + realtime_object, pool = _driven_pool() + + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCING + + +# UTS: objects/unit/RTO4b/attached-no-objects-synced-0 +async def test_rto4b_attached_no_objects_synced(): + realtime_object, pool = _driven_pool() + pool['counter:abc@1000'] = InternalLiveCounter('counter:abc@1000') + pool['root'].data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + root = pool['root'] + + updates = [] + pool['root'].subscribe(updates.append) + + _process_attached(realtime_object, {'action': ATTACHED, 'channel': 'test', 'flags': 0}) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert 'counter:abc@1000' not in pool + assert 'root' in pool + # RTO4b2: the root map's data is cleared in place; the root is never replaced + assert pool['root'] is root + assert pool['root'].data == {} + assert len(updates) >= 1 + assert updates[0].update == {'name': 'removed'} + assert updates[0].object_message is None + + +# UTS: objects/unit/RTO4b2a/reset-of-empty-root-emits-no-update-0 +async def test_rto4b2a_reset_of_empty_root_emits_no_update(): + realtime_object, pool = _driven_pool() + pool['counter:abc@1000'] = InternalLiveCounter('counter:abc@1000') + # The root is already empty, the zero value of RTLM4c + pool['root'].data = {} + + updates = [] + pool['root'].subscribe(updates.append) + + _process_attached(realtime_object, {'action': ATTACHED, 'channel': 'test', 'flags': 0}) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + # RTO4b1: the objects other than the root are still removed + assert 'counter:abc@1000' not in pool + assert 'root' in pool + assert pool['root'].data == {} + # RTO4b2a: no key was removed, so the update has no changed keys, is a no-op and is + # not delivered + assert len(updates) == 0 + + # Liveness control: a reset that does remove a key is delivered, so the count of zero + # above is the empty root's no-op and not a dead subscription. Emission is synchronous + # at this tier, so there is nothing to wait for. + realtime_object2, pool2 = _driven_pool() + pool2['root'].data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Alice'), timeserial='01', tombstone=False), + } + control = [] + pool2['root'].subscribe(control.append) + _process_attached(realtime_object2, {'action': ATTACHED, 'channel': 'test', 'flags': 0}) + assert len(control) >= 1 + assert control[0].update == {'name': 'removed'} + + +# UTS: objects/unit/RTO5/sync-complete-sequence-0 +async def test_rto5_sync_complete_sequence(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': {'name': {'data': {'string': 'Alice'}, 'timeserial': 't:0'}}, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:abc@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 42}}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert 'root' in pool + assert 'counter:abc@1000' in pool + assert pool['root'].data['name'].data == ObjectData(string='Alice') + assert pool['counter:abc@1000'].data == 42 + + +# UTS: objects/unit/RTO5a2/new-sequence-discards-old-0 +async def test_rto5a2_new_sequence_discards_old(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'seq1:cursor', flags=HAS_OBJECTS)) + _process_object_sync(realtime_object, build_object_sync_message('test', 'seq1:more', [ + build_object_state('counter:old@1000', {'aaa': 't:0'}, counter={'count': 10}), + ])) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'seq2:', [ + _empty_root_state(), + build_object_state('counter:new@1000', {'aaa': 't:0'}, counter={'count': 99}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert 'counter:old@1000' not in pool + assert 'counter:new@1000' in pool + + +# UTS: objects/unit/RTO5a5/absent-channel-serial-0 +async def test_rto5a5_absent_channel_serial(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + # No channelSerial: the whole sync is contained in this one message (RTO5a5) + _process_object_sync(realtime_object, build_object_sync_message('test', None, [ + build_object_state('counter:new@1000', {'aaa': 't:0'}, counter={'count': 99}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert 'counter:new@1000' in pool + + +# UTS: objects/unit/RTO5a6/malformed-channel-serial-treated-as-absent-0 +async def test_rto5a6_malformed_channel_serial_treated_as_absent(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + # 'malformedserialnocolon' has no ':' separator, so it cannot be parsed per RTO5a1; + # RTO5a6 handles it as if the channelSerial were absent (RTO5a5) + _process_object_sync(realtime_object, build_object_sync_message('test', 'malformedserialnocolon', [ + build_object_state('counter:new@1000', {'aaa': 't:0'}, counter={'count': 99}), + ])) + + # Treated as absent (RTO5a5): the message was applied and the sync ended + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert 'counter:new@1000' in pool + + +# UTS: objects/unit/RTO5f2a/partial-map-merge-0 +async def test_rto5f2a_partial_map_merge(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:more', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': {'name': {'data': {'string': 'Alice'}, 'timeserial': 't:0'}}, + }), + ])) + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': {'age': {'data': {'number': 30}, 'timeserial': 't:0'}}, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + ])) + + assert pool['root'].data['name'].data == ObjectData(string='Alice') + assert pool['root'].data['age'].data == ObjectData(number=30) + + +# UTS: objects/unit/RTO5c2/remove-absent-objects-0 +async def test_rto5c2_remove_absent_objects(): + realtime_object, pool = _driven_pool() + pool['counter:old@1000'] = InternalLiveCounter('counter:old@1000') + pool['counter:old@1000'].data = 99 + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [_empty_root_state()])) + + assert 'counter:old@1000' not in pool + assert 'root' in pool + + +# UTS: objects/unit/RTO5c9/clear-applied-on-ack-serials-0 +async def test_rto5c9_clear_applied_on_ack_serials(): + pool = ObjectsPool() + realtime_object = RealtimeObject(pool=pool) + realtime_object._applied_on_ack_serials = {'serial-1', 'serial-2'} + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [_empty_root_state()])) + + # The specification's `{}` is the empty set + assert realtime_object._applied_on_ack_serials == set() + + +# UTS: objects/unit/RTO8a/buffer-during-syncing-0 +async def test_rto8a_buffer_during_syncing(): + pool = ObjectsPool() + realtime_object = RealtimeObject(pool=pool) + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_message(realtime_object, build_object_message('test', [ + build_counter_inc('counter:abc@1000', 5, '01', 'site1'), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCING + assert len(realtime_object._buffered_object_operations) == 1 + assert 'counter:abc@1000' not in pool + + +# UTS: objects/unit/RTO5c6/apply-buffered-on-sync-0 +async def test_rto5c6_apply_buffered_on_sync(): + pool = ObjectsPool() + realtime_object = RealtimeObject(pool=pool) + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + _process_object_message(realtime_object, build_object_message('test', [ + build_counter_inc('counter:abc@1000', 10, '02', 'site1'), + ])) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + _empty_root_state(), + build_object_state('counter:abc@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 100}}), + ])) + + assert pool['counter:abc@1000'].data == 110 + assert len(realtime_object._buffered_object_operations) == 0 + + +# UTS: objects/unit/RTO9a1/null-operation-warning-0 +async def test_rto9a1_null_operation_warning(): + realtime_object, pool = _driven_pool() + realtime_object._sync_state = ObjectsSyncState.SYNCED + + # The specification's `ObjectMessage(serial: "01", siteCode: "site1", operation: null)` + _process_object_message(realtime_object, build_object_message('test', [ + {'serial': '01', 'siteCode': 'site1'}, + ])) + + assert len(pool) == 1 + + +# UTS: objects/unit/RTO9a3/dedup-applied-on-ack-0 +async def test_rto9a3_dedup_applied_on_ack(): + pool = ObjectsPool() + realtime_object = RealtimeObject(pool=pool) + realtime_object._sync_state = ObjectsSyncState.SYNCED + pool['counter:abc@1000'] = InternalLiveCounter('counter:abc@1000') + pool['counter:abc@1000'].data = 10 + realtime_object._applied_on_ack_serials = {'echo-serial-1'} + + _process_object_message(realtime_object, build_object_message('test', [ + { + 'serial': 'echo-serial-1', + 'siteCode': 'site1', + 'operation': { + 'action': int(ObjectOperationAction.COUNTER_INC), + 'objectId': 'counter:abc@1000', + 'counterInc': {'number': 5}, + }, + }, + ])) + + assert pool['counter:abc@1000'].data == 10 + assert 'echo-serial-1' not in realtime_object._applied_on_ack_serials + + +# UTS: objects/unit/RTO9a2a4/local-source-adds-serial-0 +async def test_rto9a2a4_local_source_adds_serial(): + pool = ObjectsPool() + realtime_object = RealtimeObject(pool=pool) + realtime_object._sync_state = ObjectsSyncState.SYNCED + pool['counter:abc@1000'] = InternalLiveCounter('counter:abc@1000') + + realtime_object._apply_object_messages([ + object_message(build_counter_inc('counter:abc@1000', 5, 'local-serial-1', 'test-site')), + ], ObjectsOperationSource.LOCAL) + + assert 'local-serial-1' in realtime_object._applied_on_ack_serials + assert pool['counter:abc@1000'].data == 5 + + +# UTS: objects/unit/RTO9a2b/unsupported-action-warning-0 +async def test_rto9a2b_unsupported_action_warning(): + realtime_object, pool = _driven_pool() + realtime_object._sync_state = ObjectsSyncState.SYNCED + + # The specification's `action: "UNKNOWN_ACTION"`: 99 is no ObjectOperationAction, so it + # decodes to UNKNOWN (OOP2a) + _process_object_message(realtime_object, build_object_message('test', [ + {'serial': '01', 'siteCode': 'site1', 'operation': {'action': 99, 'objectId': 'counter:abc@1000'}}, + ])) + + assert len(pool) == 1 + + +# UTS: objects/unit/RTO6/zero-value-from-prefix-0 +async def test_rto6_zero_value_from_prefix(): + realtime_object, pool = _driven_pool() + realtime_object._sync_state = ObjectsSyncState.SYNCED + + _process_object_message(realtime_object, build_object_message('test', [ + build_counter_inc('counter:new@2000', 5, '01', 'site1'), + ])) + _process_object_message(realtime_object, build_object_message('test', [ + build_map_set('map:new@2000', 'key', {'string': 'val'}, '01', 'site1'), + ])) + + assert 'counter:new@2000' in pool + assert isinstance(pool['counter:new@2000'], InternalLiveCounter) + assert pool['counter:new@2000'].data == 5 + + assert 'map:new@2000' in pool + assert isinstance(pool['map:new@2000'], InternalLiveMap) + assert pool['map:new@2000'].data['key'].data == ObjectData(string='val') + + +# UTS: objects/unit/RTO5d/null-object-skipped-0 +async def test_rto5d_null_object_skipped(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + # The specification's `ObjectMessage(object: null)` is the empty message `{}` + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + {}, + _empty_root_state(), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + + +# UTS: objects/unit/RTO5f3/unsupported-type-skipped-0 +async def test_rto5f3_unsupported_type_skipped(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + _empty_root_state(), + {'object': {'objectId': 'unknown:xyz@1000', 'siteTimeserials': {}}}, + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert 'unknown:xyz@1000' not in pool + + +# UTS: objects/unit/RTO5e/object-sync-transitions-syncing-0 +async def test_rto5e_object_sync_transitions_syncing(): + realtime_object, pool = _driven_pool() + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:more', [ + build_object_state('root', {'aaa': 't:0'}, map={'semantics': LWW, 'entries': {}}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCING + + +# UTS: objects/unit/RTO5c7/sync-emits-updates-0 +async def test_rto5c7_sync_emits_updates(): + realtime_object, pool = _driven_pool() + pool['root'].data = { + 'name': ObjectsMapEntry(data=ObjectData(string='Old'), timeserial='01', tombstone=False), + } + + updates = [] + pool['root'].subscribe(updates.append) + + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': {'name': {'data': {'string': 'New'}, 'timeserial': 't:0'}}, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + ])) + + assert len(updates) >= 1 + assert 'name' in updates[0].update + assert updates[0].update['name'] == 'updated' + + +# UTS: objects/unit/RTO5f2b/partial-counter-error-0 +async def test_rto5f2b_partial_counter_error(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:more', [ + build_object_state('counter:abc@1000', {'aaa': 't:0'}, counter={'count': 10}), + ])) + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + _empty_root_state(), + build_object_state('counter:abc@1000', {'aaa': 't:0'}, counter={'count': 5}), + ])) + + assert pool['counter:abc@1000'].data == 10 + + +# UTS: objects/unit/RTO4d/attached-clears-buffer-0 +async def test_rto4d_attached_clears_buffer(): + pool = ObjectsPool() + realtime_object = RealtimeObject(pool=pool) + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + _process_object_message(realtime_object, build_object_message('test', [ + build_counter_inc('counter:abc@1000', 5, '01', 'site1'), + ])) + assert len(realtime_object._buffered_object_operations) == 1 + + _process_attached(realtime_object, objects_attached_message('test', 'sync2:cursor', flags=HAS_OBJECTS)) + + assert len(realtime_object._buffered_object_operations) == 0 + + +# UTS: objects/unit/RTO4-RTO5/attached-during-syncing-resets-0 +async def test_rto4_rto5_attached_during_syncing_resets(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:more', [ + build_object_state('counter:old@1000', {'aaa': 't:0'}, counter={'count': 10}), + ])) + assert realtime_object._sync_state == ObjectsSyncState.SYNCING + + _process_attached(realtime_object, objects_attached_message('test', 'sync2:cursor', flags=HAS_OBJECTS)) + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync2:', [ + _empty_root_state(), + build_object_state('counter:new@1000', {'aaa': 't:0'}, counter={'count': 99}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert 'counter:old@1000' not in pool + assert 'counter:new@1000' in pool + + +# UTS: objects/unit/RTO5-RTO7/new-sync-keeps-buffer-0 +async def test_rto5_rto7_new_sync_keeps_buffer(): + pool = ObjectsPool() + realtime_object = RealtimeObject(pool=pool) + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + _process_object_message(realtime_object, build_object_message('test', [ + build_counter_inc('counter:abc@1000', 5, '01', 'site1'), + ])) + assert len(realtime_object._buffered_object_operations) == 1 + + _process_object_sync(realtime_object, build_object_sync_message('test', 'seq2:', [ + _empty_root_state(), + build_object_state('counter:abc@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 100}}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert pool['counter:abc@1000'].data == 105 + + +# UTS: objects/unit/RTO7-RTO8/buffer-without-attached-0 +async def test_rto7_rto8_buffer_without_attached(): + pool = ObjectsPool() + realtime_object = RealtimeObject(pool=pool) + assert realtime_object._sync_state == ObjectsSyncState.INITIALIZED + + _process_object_message(realtime_object, build_object_message('test', [ + build_counter_inc('counter:abc@1000', 5, '01', 'site1'), + ])) + + assert len(realtime_object._buffered_object_operations) == 1 + + +# UTS: objects/unit/RTO5c-RTLM23/sync-clear-timeserial-hides-create-entries-0 +async def test_rto5c_rtlm23_sync_clear_timeserial_hides_create_entries(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': {}, + 'clearTimeserial': '05', + }, create_op={'mapCreate': { + 'semantics': LWW, + 'entries': { + 'old_key': {'data': {'string': 'old'}, 'timeserial': '03'}, + 'new_key': {'data': {'string': 'new'}, 'timeserial': '07'}, + }, + }}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + assert 'old_key' not in pool['root'].data + assert pool['root'].data['new_key'].data == ObjectData(string='new') + + +# UTS: objects/unit/RTO5c10/sync-rebuilds-parent-refs-0 +async def test_rto5c10_sync_rebuilds_parent_refs(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': { + 'score': {'data': {'objectId': 'counter:score@1000'}, 'timeserial': 't:0'}, + 'profile': {'data': {'objectId': 'map:profile@1000'}, 'timeserial': 't:0'}, + 'name': {'data': {'string': 'Alice'}, 'timeserial': 't:0'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:score@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 100}}), + build_object_state('map:profile@1000', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': { + 'nested_counter': {'data': {'objectId': 'counter:nested@1000'}, 'timeserial': 't:0'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:nested@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 5}}), + ])) + + # root is not referenced by any parent + assert pool['root'].parent_references == {} + # counter:score@1000 is referenced by root at key 'score' + assert pool['counter:score@1000'].parent_references == {'root': {'score'}} + # map:profile@1000 is referenced by root at key 'profile' + assert pool['map:profile@1000'].parent_references == {'root': {'profile'}} + # counter:nested@1000 is referenced by map:profile@1000 at key 'nested_counter' + assert pool['counter:nested@1000'].parent_references == {'map:profile@1000': {'nested_counter'}} + # The primitive-valued entry 'name' appears in no parent_references, which the + # equalities above already establish + + +# UTS: objects/unit/RTO5c10/resync-rebuilds-parent-refs-0 +async def test_rto5c10_resync_rebuilds_parent_refs(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + # First sync: counter:abc@1000 is a child of root + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': { + 'counter_key': {'data': {'objectId': 'counter:abc@1000'}, 'timeserial': 't:0'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:abc@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 10}}), + ])) + assert pool['counter:abc@1000'].parent_references == {'root': {'counter_key'}} + + # Second sync: counter:abc@1000 is a child of map:wrapper@1000, not of root + _process_attached(realtime_object, objects_attached_message('test', 'sync2:cursor', flags=HAS_OBJECTS)) + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync2:', [ + build_object_state('root', {'aaa': 't:1'}, map={ + 'semantics': LWW, + 'entries': { + 'wrapper': {'data': {'objectId': 'map:wrapper@1000'}, 'timeserial': 't:1'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('map:wrapper@1000', {'aaa': 't:1'}, map={ + 'semantics': LWW, + 'entries': { + 'moved_counter': {'data': {'objectId': 'counter:abc@1000'}, 'timeserial': 't:1'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:abc@1000', {'aaa': 't:1'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 20}}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + # root is not referenced by any parent + assert pool['root'].parent_references == {} + # map:wrapper@1000 is a child of root at key 'wrapper' + assert pool['map:wrapper@1000'].parent_references == {'root': {'wrapper'}} + # counter:abc@1000 is a child of map:wrapper@1000, and no longer of root + assert pool['counter:abc@1000'].parent_references == {'map:wrapper@1000': {'moved_counter'}} + + +# UTS: objects/unit/RTO5c10/empty-sync-parent-refs-0 +async def test_rto5c10_empty_sync_parent_refs(): + realtime_object, pool = _driven_pool() + + # A normal sync first, to populate parent_references + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': { + 'child': {'data': {'objectId': 'counter:child@1000'}, 'timeserial': 't:0'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:child@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 1}}), + ])) + assert pool['counter:child@1000'].parent_references == {'root': {'child'}} + + # An empty sync: ATTACHED without HAS_OBJECTS + _process_attached(realtime_object, {'action': ATTACHED, 'channel': 'test', 'flags': 0}) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + # counter:child@1000 was removed from the pool (RTO4b1) + assert 'counter:child@1000' not in pool + # root remains, with empty data and empty parent_references + assert 'root' in pool + assert pool['root'].data == {} + assert pool['root'].parent_references == {} diff --git a/test/uts/objects/unit/parent_references_test.py b/test/uts/objects/unit/parent_references_test.py new file mode 100644 index 00000000..f7077ee3 --- /dev/null +++ b/test/uts/objects/unit/parent_references_test.py @@ -0,0 +1,433 @@ +"""Derived from uts/objects/unit/parent_references.md in ably/specification. + +Spec points: RTLO3f, RTLO3f2, RTLO4g, RTLO4g1, RTLO4g2, RTLO4h, RTLO4h1, RTLO4h2, RTLO4h3, +RTLO4f, RTLO4f1, RTLO4f2, RTLO4f3, RTLO4f4, RTO5c10, RTO5c10a, RTO5c10b + +A pure unit specification: nothing is mocked and nothing is connected. Objects are built on +their own and then put in a pool with `pool[object_id] = obj`, which adopts them, so +`get_full_paths()` resolves their parents' ids through that pool. A path is a list of keys, +so the specification's `paths CONTAINS ["a", "x"]` is `['a', 'x'] in paths`. + +The three RTO5c10 cases drive a sync, which the specification does through +`ObjectsPool.processAttached` and `processObjectSync`. The sync state machine is +`RealtimeObject`'s (shape deviation S-2), so those cases work on the pool a standalone +`RealtimeObject()` holds and drive the `RealtimeObject`, through the helpers below; they are +coroutines only so that an implementation is free to touch the event loop. +""" + +from ably.pubsub.liveobjects.enums import ObjectsSyncState +from ably.pubsub.liveobjects.livecounter import InternalLiveCounter +from ably.pubsub.liveobjects.livemap import InternalLiveMap +from ably.pubsub.liveobjects.objectmessage import ObjectsMapSemantics +from ably.pubsub.liveobjects.objectspool import ObjectsPool +from ably.pubsub.liveobjects.realtimeobject import RealtimeObject +from test.uts.objects.helpers.standard_test_pool import ( + HAS_OBJECTS, + LWW, + build_object_state, + build_object_sync_message, + object_messages, + objects_attached_message, +) + + +def _driven_pool(): + """The specification's `pool = ObjectsPool()`, in a test that then drives the sync state machine. + + S-2: the state machine is `RealtimeObject`'s, so the pool is the one a standalone + `RealtimeObject` holds. Returns `(realtime_object, pool)`. + """ + realtime_object = RealtimeObject() + return realtime_object, realtime_object._objects_pool + + +def _process_attached(realtime_object, attached): + """The specification's `pool.processAttached(attached)` (S-2). + + `RealtimeObject` is given the ATTACHED's HAS_OBJECTS flag and nothing else (RTO4). + """ + realtime_object._on_attached(has_objects=bool(attached.get('flags', 0) & HAS_OBJECTS)) + + +def _process_object_sync(realtime_object, protocol_message): + """The specification's `pool.processObjectSync(protocol_message)` (S-2).""" + realtime_object._handle_object_sync_messages( + object_messages(protocol_message), protocol_message.get('channelSerial')) + + +# UTS: objects/unit/RTLO3f2/init-empty-counter-0 +def test_rtlo3f2_init_empty_counter(): + counter = InternalLiveCounter('counter:abc@1000') + + assert counter.parent_references == {} + + +# UTS: objects/unit/RTLO3f2/init-empty-map-0 +def test_rtlo3f2_init_empty_map(): + live_map = InternalLiveMap('map:abc@1000', ObjectsMapSemantics.LWW) + + assert live_map.parent_references == {} + + +# UTS: objects/unit/RTLO4g2/first-reference-new-entry-0 +def test_rtlo4g2_first_reference_new_entry(): + child = InternalLiveCounter('counter:child@1000') + parent = InternalLiveMap('map:parent@1000', ObjectsMapSemantics.LWW) + + child.add_parent_reference(parent, 'score') + + assert 'map:parent@1000' in child.parent_references + assert child.parent_references['map:parent@1000'] == {'score'} + + +# UTS: objects/unit/RTLO4g1/second-key-same-parent-0 +def test_rtlo4g1_second_key_same_parent(): + child = InternalLiveCounter('counter:child@1000') + parent = InternalLiveMap('map:parent@1000', ObjectsMapSemantics.LWW) + child.parent_references = {'map:parent@1000': {'score'}} + + child.add_parent_reference(parent, 'points') + + assert child.parent_references['map:parent@1000'] == {'score', 'points'} + + +# UTS: objects/unit/RTLO4g/different-parent-separate-entry-0 +def test_rtlo4g_different_parent_separate_entry(): + child = InternalLiveCounter('counter:child@1000') + parent_a = InternalLiveMap('map:a@1000', ObjectsMapSemantics.LWW) + parent_b = InternalLiveMap('map:b@1000', ObjectsMapSemantics.LWW) + + child.add_parent_reference(parent_a, 'x') + child.add_parent_reference(parent_b, 'y') + + assert child.parent_references['map:a@1000'] == {'x'} + assert child.parent_references['map:b@1000'] == {'y'} + + +# UTS: objects/unit/RTLO4g/multiple-parents-multiple-keys-0 +def test_rtlo4g_multiple_parents_multiple_keys(): + child = InternalLiveCounter('counter:child@1000') + parent_a = InternalLiveMap('map:a@1000', ObjectsMapSemantics.LWW) + parent_b = InternalLiveMap('map:b@1000', ObjectsMapSemantics.LWW) + + child.add_parent_reference(parent_a, 'x') + child.add_parent_reference(parent_a, 'y') + child.add_parent_reference(parent_b, 'p') + child.add_parent_reference(parent_b, 'q') + + assert child.parent_references['map:a@1000'] == {'x', 'y'} + assert child.parent_references['map:b@1000'] == {'p', 'q'} + + +# UTS: objects/unit/RTLO4h1/nonexistent-parent-noop-0 +def test_rtlo4h1_nonexistent_parent_noop(): + child = InternalLiveCounter('counter:child@1000') + parent = InternalLiveMap('map:parent@1000', ObjectsMapSemantics.LWW) + + child.remove_parent_reference(parent, 'score') + + assert child.parent_references == {} + + +# UTS: objects/unit/RTLO4h2/remove-key-leaves-others-0 +def test_rtlo4h2_remove_key_leaves_others(): + child = InternalLiveCounter('counter:child@1000') + parent = InternalLiveMap('map:parent@1000', ObjectsMapSemantics.LWW) + child.parent_references = {'map:parent@1000': {'score', 'points'}} + + child.remove_parent_reference(parent, 'score') + + assert child.parent_references['map:parent@1000'] == {'points'} + + +# UTS: objects/unit/RTLO4h3/remove-last-key-removes-entry-0 +def test_rtlo4h3_remove_last_key_removes_entry(): + child = InternalLiveCounter('counter:child@1000') + parent = InternalLiveMap('map:parent@1000', ObjectsMapSemantics.LWW) + child.parent_references = {'map:parent@1000': {'score'}} + + child.remove_parent_reference(parent, 'score') + + assert 'map:parent@1000' not in child.parent_references + assert child.parent_references == {} + + +# UTS: objects/unit/RTLO4h/remove-nonexistent-key-0 +def test_rtlo4h_remove_nonexistent_key(): + child = InternalLiveCounter('counter:child@1000') + parent = InternalLiveMap('map:parent@1000', ObjectsMapSemantics.LWW) + child.parent_references = {'map:parent@1000': {'score'}} + + child.remove_parent_reference(parent, 'nonexistent') + + assert child.parent_references['map:parent@1000'] == {'score'} + + +# UTS: objects/unit/RTLO4f2/root-returns-empty-path-0 +def test_rtlo4f2_root_returns_empty_path(): + pool = ObjectsPool() + root = pool['root'] + + paths = root.get_full_paths() + assert len(paths) == 1 + assert [] in paths + + +# UTS: objects/unit/RTLO4f/direct-child-single-path-0 +def test_rtlo4f_direct_child_single_path(): + pool = ObjectsPool() + counter = InternalLiveCounter('counter:score@1000') + pool['counter:score@1000'] = counter + + root = pool['root'] + counter.add_parent_reference(root, 'score') + + paths = counter.get_full_paths() + assert len(paths) == 1 + assert ['score'] in paths + + +# UTS: objects/unit/RTLO4f/deep-nesting-0 +def test_rtlo4f_deep_nesting(): + # root --'profile'--> map:profile --'prefs'--> map:prefs --'theme_counter'--> counter:theme + pool = ObjectsPool() + root = pool['root'] + + profile = InternalLiveMap('map:profile@1000', ObjectsMapSemantics.LWW) + pool['map:profile@1000'] = profile + profile.add_parent_reference(root, 'profile') + + prefs = InternalLiveMap('map:prefs@1000', ObjectsMapSemantics.LWW) + pool['map:prefs@1000'] = prefs + prefs.add_parent_reference(profile, 'prefs') + + theme_counter = InternalLiveCounter('counter:theme@1000') + pool['counter:theme@1000'] = theme_counter + theme_counter.add_parent_reference(prefs, 'theme_counter') + + paths = theme_counter.get_full_paths() + assert len(paths) == 1 + assert ['profile', 'prefs', 'theme_counter'] in paths + + +# UTS: objects/unit/RTLO4f/diamond-graph-0 +def test_rtlo4f_diamond_graph(): + # root --'a'--> map:A --'x'--> counter:leaf, and root --'b'--> map:B --'y'--> counter:leaf + pool = ObjectsPool() + root = pool['root'] + + map_a = InternalLiveMap('map:a@1000', ObjectsMapSemantics.LWW) + pool['map:a@1000'] = map_a + map_a.add_parent_reference(root, 'a') + + map_b = InternalLiveMap('map:b@1000', ObjectsMapSemantics.LWW) + pool['map:b@1000'] = map_b + map_b.add_parent_reference(root, 'b') + + leaf = InternalLiveCounter('counter:leaf@1000') + pool['counter:leaf@1000'] = leaf + leaf.add_parent_reference(map_a, 'x') + leaf.add_parent_reference(map_b, 'y') + + paths = leaf.get_full_paths() + assert len(paths) == 2 + assert ['a', 'x'] in paths + assert ['b', 'y'] in paths + + +# UTS: objects/unit/RTLO4f/single-parent-multiple-keys-0 +def test_rtlo4f_single_parent_multiple_keys(): + pool = ObjectsPool() + root = pool['root'] + + child = InternalLiveCounter('counter:child@1000') + pool['counter:child@1000'] = child + child.add_parent_reference(root, 'primary') + child.add_parent_reference(root, 'alias') + + paths = child.get_full_paths() + assert len(paths) == 2 + assert ['primary'] in paths + assert ['alias'] in paths + + +# UTS: objects/unit/RTLO4f/orphan-returns-empty-0 +def test_rtlo4f_orphan_returns_empty(): + pool = ObjectsPool() + + orphan = InternalLiveCounter('counter:orphan@1000') + pool['counter:orphan@1000'] = orphan + + paths = orphan.get_full_paths() + assert len(paths) == 0 + + +# UTS: objects/unit/RTLO4f/cycle-suppression-0 +def test_rtlo4f_cycle_suppression(): + # root --'a'--> map:A --'b'--> map:B --'a'--> map:A, a cycle. The only simple path to + # map:B is ['a', 'b']. + pool = ObjectsPool() + root = pool['root'] + + map_a = InternalLiveMap('map:a@1000', ObjectsMapSemantics.LWW) + pool['map:a@1000'] = map_a + map_a.add_parent_reference(root, 'a') + + map_b = InternalLiveMap('map:b@1000', ObjectsMapSemantics.LWW) + pool['map:b@1000'] = map_b + map_b.add_parent_reference(map_a, 'b') + + # The cycle: map:A also has map:B as a parent + map_a.add_parent_reference(map_b, 'a') + + paths_b = map_b.get_full_paths() + assert len(paths_b) == 1 + assert ['a', 'b'] in paths_b + + paths_a = map_a.get_full_paths() + assert len(paths_a) == 1 + assert ['a'] in paths_a + + +# UTS: objects/unit/RTLO4f/complex-diamond-deep-0 +def test_rtlo4f_complex_diamond_deep(): + # root --'left'--> map:L --'mid'--> map:M --'target'--> counter:T, and + # root --'right'--> map:R --'target'--> counter:T + pool = ObjectsPool() + root = pool['root'] + + map_l = InternalLiveMap('map:l@1000', ObjectsMapSemantics.LWW) + pool['map:l@1000'] = map_l + map_l.add_parent_reference(root, 'left') + + map_r = InternalLiveMap('map:r@1000', ObjectsMapSemantics.LWW) + pool['map:r@1000'] = map_r + map_r.add_parent_reference(root, 'right') + + map_m = InternalLiveMap('map:m@1000', ObjectsMapSemantics.LWW) + pool['map:m@1000'] = map_m + map_m.add_parent_reference(map_l, 'mid') + + target = InternalLiveCounter('counter:t@1000') + pool['counter:t@1000'] = target + target.add_parent_reference(map_m, 'target') + target.add_parent_reference(map_r, 'target') + + paths = target.get_full_paths() + assert len(paths) == 2 + assert ['left', 'mid', 'target'] in paths + assert ['right', 'target'] in paths + + +# UTS: objects/unit/RTO5c10/rebuild-from-sync-0 +async def test_rto5c10_rebuild_from_sync(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': { + 'score': {'data': {'objectId': 'counter:score@1000'}, 'timeserial': 't:0'}, + 'profile': {'data': {'objectId': 'map:profile@1000'}, 'timeserial': 't:0'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:score@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 100}}), + build_object_state('map:profile@1000', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': { + 'nested': {'data': {'objectId': 'counter:nested@1000'}, 'timeserial': 't:0'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:nested@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 5}}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + + # counter:score@1000 is referenced by root at key 'score' + score = pool['counter:score@1000'] + assert score.parent_references['root'] == {'score'} + + # map:profile@1000 is referenced by root at key 'profile' + profile = pool['map:profile@1000'] + assert profile.parent_references['root'] == {'profile'} + + # counter:nested@1000 is referenced by map:profile@1000 at key 'nested' + nested = pool['counter:nested@1000'] + assert nested.parent_references['map:profile@1000'] == {'nested'} + + # root has no parent references + assert pool['root'].parent_references == {} + + # get_full_paths resolves through the rebuilt references + assert ['score'] in score.get_full_paths() + assert ['profile', 'nested'] in nested.get_full_paths() + + +# UTS: objects/unit/RTO5c10a/rebuild-clears-stale-refs-0 +async def test_rto5c10a_rebuild_clears_stale_refs(): + realtime_object, pool = _driven_pool() + + # First sync: root --'score'--> counter:abc@1000 + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': { + 'score': {'data': {'objectId': 'counter:abc@1000'}, 'timeserial': 't:0'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:abc@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 10}}), + ])) + assert pool['counter:abc@1000'].parent_references['root'] == {'score'} + + # Second sync: root --'points'--> counter:abc@1000, the key changed from 'score' to 'points' + _process_attached(realtime_object, objects_attached_message('test', 'sync2:cursor', flags=HAS_OBJECTS)) + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync2:', [ + build_object_state('root', {'aaa': 't:1'}, map={ + 'semantics': LWW, + 'entries': { + 'points': {'data': {'objectId': 'counter:abc@1000'}, 'timeserial': 't:1'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:abc@1000', {'aaa': 't:1'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 20}}), + ])) + + counter = pool['counter:abc@1000'] + + # The 'score' reference is gone, replaced by 'points' + assert counter.parent_references['root'] == {'points'} + assert ['points'] in counter.get_full_paths() + + paths = counter.get_full_paths() + assert len(paths) == 1 + + +# UTS: objects/unit/RTO5c10/unreferenced-empty-refs-0 +async def test_rto5c10_unreferenced_empty_refs(): + realtime_object, pool = _driven_pool() + _process_attached(realtime_object, objects_attached_message('test', 'sync1:cursor', flags=HAS_OBJECTS)) + + _process_object_sync(realtime_object, build_object_sync_message('test', 'sync1:', [ + build_object_state('root', {'aaa': 't:0'}, map={ + 'semantics': LWW, + 'entries': { + 'name': {'data': {'string': 'Alice'}, 'timeserial': 't:0'}, + }, + }, create_op={'mapCreate': {'semantics': LWW, 'entries': {}}}), + build_object_state('counter:orphan@1000', {'aaa': 't:0'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 42}}), + ])) + + assert realtime_object._sync_state == ObjectsSyncState.SYNCED + + # The counter is in the pool, but no InternalLiveMap entry refers to it + orphan = pool['counter:orphan@1000'] + assert orphan.parent_references == {} + + # get_full_paths returns no key-path for an unreferenced object + assert len(orphan.get_full_paths()) == 0 diff --git a/test/uts/objects/unit/path_object_mutations_test.py b/test/uts/objects/unit/path_object_mutations_test.py new file mode 100644 index 00000000..0f62f201 --- /dev/null +++ b/test/uts/objects/unit/path_object_mutations_test.py @@ -0,0 +1,152 @@ +"""Derived from uts/objects/unit/path_object_mutations.md in ably/specification. + +Spec points: RTPO3c2, RTPO15, RTPO15a2, RTPO15b, RTPO15c, RTPO15d, RTPO15e, RTPO16, RTPO16b, RTPO16c, +RTPO16d, RTPO16e, RTPO17, RTPO17a1, RTPO17b, RTPO17c, RTPO17d, RTPO17e, RTPO18, RTPO18a1, RTPO18b, +RTPO18c, RTPO18d, RTPO18e, RTTS5d2 + +The specification writes through the untyped `PathObject`. ably-python partitions it (LODR-061, +RTTS3-RTTS6): `set` and `remove` are on `LiveMapPathObject` and `increment` and `decrement` on +`LiveCounterPathObject`, reached through the unchecked view helpers. `root` is already a +`LiveMapPathObject`, so `root.set(...)` needs no view; `root.get('score').increment(25)` is +`root.get('score').as_live_counter().increment(25)`. A view never raises for the type at the path; +the write through it raises 92007 for a path resolving to the wrong type and 92005 for one that does +not resolve (RTTS5d2), which is what the specification's failure cases assert. + +Every write is awaited against the standard mock, which ACKs each OBJECT message, so the operation +has been applied locally (RTO20) by the time the write returns and the read that follows needs no +wait. Reads go through the view matching the type written: `as_primitive()` for map entries, +`as_live_counter()` for the counter. +""" + +import pytest + +from ably.pubsub.util.exceptions import AblyException +from test.uts.objects.helpers.standard_test_pool import setup_synced_channel + + +# UTS: objects/unit/RTPO15/set-delegates-to-map-0 +async def test_rtpo15_set_delegates_to_map(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.set('name', 'Bob') + + assert root.get('name').as_primitive().value() == 'Bob' + + +# UTS: objects/unit/RTPO15/set-nested-path-0 +async def test_rtpo15_set_nested_path(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('profile').as_live_map().set('email', 'bob@example.com') + + assert root.get('profile').get('email').as_primitive().value() == 'bob@example.com' + + +# UTS: objects/unit/RTPO15d/set-non-map-throws-0 +async def test_rtpo15d_set_non_map_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + # The map view is unchecked (RTTS5d); `set` through it finds a counter at the path + with pytest.raises(AblyException) as excinfo: + await root.get('score').as_live_map().set('key', 'value') + + assert excinfo.value.code == 92007 + + +# UTS: objects/unit/RTPO16/remove-delegates-to-map-0 +async def test_rtpo16_remove_delegates_to_map(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.remove('name') + + assert root.get('name').as_primitive().value() is None + + +# UTS: objects/unit/RTPO16d/remove-non-map-throws-0 +async def test_rtpo16d_remove_non_map_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + await root.get('score').as_live_map().remove('key') + + assert excinfo.value.code == 92007 + + +# UTS: objects/unit/RTPO17/increment-delegates-to-counter-0 +async def test_rtpo17_increment_delegates_to_counter(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().increment(25) + + assert root.get('score').as_live_counter().value() == 125 + + +# UTS: objects/unit/RTPO17/increment-default-amount-0 +async def test_rtpo17_increment_default_amount(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().increment() + + assert root.get('score').as_live_counter().value() == 101 + + +# UTS: objects/unit/RTPO17d/increment-non-counter-throws-0 +async def test_rtpo17d_increment_non_counter_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + # `root` is a `LiveMapPathObject` and has no `increment`; the counter view over the root + # path is the translation, and the root map is the wrong type for it + with pytest.raises(AblyException) as excinfo: + await root.as_live_counter().increment(5) + + assert excinfo.value.code == 92007 + + +# UTS: objects/unit/RTPO18/decrement-delegates-to-counter-0 +async def test_rtpo18_decrement_delegates_to_counter(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().decrement(10) + + assert root.get('score').as_live_counter().value() == 90 + + +# UTS: objects/unit/RTPO18/decrement-default-amount-0 +async def test_rtpo18_decrement_default_amount(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().decrement() + + assert root.get('score').as_live_counter().value() == 99 + + +# UTS: objects/unit/RTPO18d/decrement-non-counter-throws-0 +async def test_rtpo18d_decrement_non_counter_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + await root.as_live_counter().decrement(5) + + assert excinfo.value.code == 92007 + + +# UTS: objects/unit/RTPO3c2/set-unresolvable-throws-0 +async def test_rtpo3c2_set_unresolvable_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + await root.get('nonexistent').get('deep').as_live_map().set('key', 'value') + + assert excinfo.value.code == 92005 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTPO3c2/increment-unresolvable-throws-0 +async def test_rtpo3c2_increment_unresolvable_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + await root.get('nonexistent').as_live_counter().increment(5) + + assert excinfo.value.code == 92005 + assert excinfo.value.status_code == 400 diff --git a/test/uts/objects/unit/path_object_subscribe_test.py b/test/uts/objects/unit/path_object_subscribe_test.py new file mode 100644 index 00000000..bdcb5603 --- /dev/null +++ b/test/uts/objects/unit/path_object_subscribe_test.py @@ -0,0 +1,553 @@ +"""Derived from uts/objects/unit/path_object_subscribe.md in ably/specification. + +Spec points: RTPO19, RTPO19b, RTPO19c1, RTPO19c1a, RTPO19d, RTPO19e1, RTPO19e2, RTPO19f, RTPO19g, +RTO24b1, RTO24b2, RTO24b2a, RTO24b2a1, RTO24b2a2, RTO24b2b, RTO24b2b2, RTO24b2c, RTO24c1, RTO24c2a, +RTO24c2b, RTO24c2c, RTO24c2d, RTO25b, RTLM24 + +`subscribe` is on the base `PathObject` (LODR-061, RTTS3d) and is synchronous; the specification's +`{ depth: n }` option is the keyword argument `depth=n`. The listener receives a +`PathObjectSubscriptionEvent` whose `object` is a base `PathObject`, so the counter value RTPO19e1 +reads is `event.object.as_live_counter().value()`, and whose `message` is the public `ObjectMessage` +with snake_case fields (`site_code`, `operation.object_id`, `operation.counter_inc`) and the action as +an `ObjectOperationAction`. + +Each stimulus is an inbound OBJECT or OBJECT_SYNC message, which the client applies on the +transport's read task, so every positive expectation is reached with `poll_until`. Where the +specification then asserts an exact count, or that a listener stayed silent, the test first awaits +the delivery of a control (a further event, or an unlimited-depth listener on the same dispatch) and +settles, so that a late or spurious callback would already have run. Where the specification +subscribes straight after sending a seeding message, the test waits for the seed to be applied +first, since a seed applied after the subscription would itself be delivered to it. +""" + +import pytest + +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction +from ably.pubsub.liveobjects.pathobject import PathObject +from ably.pubsub.liveobjects.subscription import Subscription +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import await_channel_state, poll_until +from test.uts.helpers.clock import settle +from test.uts.objects.helpers.standard_test_pool import ( + build_counter_inc, + build_map_clear, + build_map_set, + build_object_message, + build_object_state, + build_object_sync_message, + objects_channel_options, + objects_client, + remote_serial, + setup_synced_channel, + standard_mock_websocket, +) + + +# UTS: objects/unit/RTPO19/subscribe-receives-events-0 +async def test_rtpo19_subscribe_receives_events(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + sub = root.get('score').subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the COUNTER_INC event') + await settle() + + assert isinstance(sub, Subscription) + assert len(events) == 1 + assert isinstance(events[0].object, PathObject) + assert events[0].object.path() == 'score' + assert events[0].message is not None + assert events[0].message.serial == '99' + assert events[0].message.site_code == 'remote' + assert events[0].message.operation is not None + assert events[0].message.operation.action == ObjectOperationAction.COUNTER_INC + assert events[0].message.channel == 'test' + + +# UTS: objects/unit/RTPO19b/subscribe-precondition-detached-0 +async def test_rtpo19b_subscribe_precondition_detached(): + # The specification's hand-written mock is the standard one: CONNECTED with siteCode and + # objectsGCGracePeriod, ATTACH answered with ATTACHED and the standard pool's OBJECT_SYNC, + # DETACH answered with DETACHED + mock_ws = standard_mock_websocket() + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options()) + root = await channel.object.get() + + await channel.detach() + await await_channel_state(channel, ChannelState.DETACHED) + + with pytest.raises(AblyException) as excinfo: + root.subscribe(lambda event: None) + + assert excinfo.value.code == 90001 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTPO19c1a/subscribe-non-positive-depth-throws-0 +async def test_rtpo19c1a_subscribe_non_positive_depth_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + root.subscribe(lambda event: None, depth=0) + + assert excinfo.value.code == 40003 + + +# UTS: objects/unit/RTPO19c1a/subscribe-negative-depth-throws-0 +async def test_rtpo19c1a_subscribe_negative_depth_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + root.subscribe(lambda event: None, depth=-1) + + assert excinfo.value.code == 40003 + + +# UTS: objects/unit/RTPO19c1/subscribe-depth-1-self-only-0 +async def test_rtpo19c1_subscribe_depth_1_self_only(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.subscribe(events.append, depth=1) + # Quiescence control: an unlimited-depth root listener covers the out-of-scope child path + control = [] + root.subscribe(control.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the self event') + await settle() + + control_before = len(control) + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '100', 'remote'), + ])) + await poll_until(lambda: len(control) > control_before, description='the control to see the child event') + await settle() + + assert len(events) == 1 + + +# UTS: objects/unit/RTPO19c1/subscribe-depth-2-children-0 +async def test_rtpo19c1_subscribe_depth_2_children(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.subscribe(events.append, depth=2) + # Quiescence control: an unlimited-depth root listener covers the out-of-scope grandchild path + control = [] + root.subscribe(control.append) + + # Self event (root map update): the candidate [] is covered at depth 2 + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the self event') + + # Child event (the counter at ["score"]): relative depth 1 - 0 + 1 = 2 <= 2, covered + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '100', 'remote'), + ])) + await poll_until(lambda: len(events) >= 2, description='the child event') + await settle() + + # Grandchild event (the counter at ["profile", "nested_counter"]): relative depth + # 2 - 0 + 1 = 3 > 2, not covered. A COUNTER_INC yields this single candidate only. + control_before = len(control) + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:nested@1000', 1, '101', 'remote'), + ])) + await poll_until(lambda: len(control) > control_before, description='the control to see the grandchild event') + await settle() + + assert len(events) == 2 + + +# UTS: objects/unit/RTPO19c1/subscribe-unlimited-depth-0 +async def test_rtpo19c1_subscribe_unlimited_depth(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the root event') + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '100', 'remote'), + ])) + await poll_until(lambda: len(events) >= 2, description='the child event') + + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('map:prefs@1000', 'theme', {'string': 'light'}, remote_serial(1), 'remote'), + ])) + await poll_until(lambda: len(events) >= 3, description='the descendant event') + + assert len(events) >= 3 + + +# UTS: objects/unit/RTPO19d/subscribe-returns-subscription-0 +async def test_rtpo19d_subscribe_returns_subscription(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + sub = root.get('score').subscribe(events.append) + # Quiescence control: a separate listener at the same path, which stays subscribed + control = [] + root.get('score').subscribe(control.append) + + assert isinstance(sub, Subscription) + sub.unsubscribe() + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote'), + ])) + await poll_until(lambda: len(control) >= 1, description='the control listener to fire') + await settle() + + assert len(events) == 0 + + +# UTS: objects/unit/RTPO19e1/event-path-object-correct-0 +async def test_rtpo19e1_event_path_object_correct(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the COUNTER_INC event') + + assert isinstance(events[0].object, PathObject) + assert events[0].object.path() == 'score' + assert events[0].object.as_live_counter().value() == 107 + + +# UTS: objects/unit/RTPO19e2/event-message-delivery-0 +async def test_rtpo19e2_event_message_delivery(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.get('score').subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 42, 'serial-1', 'site-a'), + ])) + await poll_until(lambda: len(events) >= 1, description='the COUNTER_INC event') + + assert events[0].message is not None + assert events[0].message.channel == 'test' + assert events[0].message.serial == 'serial-1' + assert events[0].message.site_code == 'site-a' + assert events[0].message.operation is not None + assert events[0].message.operation.action == ObjectOperationAction.COUNTER_INC + assert events[0].message.operation.object_id == 'counter:score@1000' + assert events[0].message.operation.counter_inc.number == 42 + + +# UTS: objects/unit/RTPO19e2/event-message-omitted-no-operation-0 +async def test_rtpo19e2_event_message_omitted_no_operation(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.subscribe(events.append) + + # An OBJECT_SYNC that moves counter:score@1000 from 100 to 200 through replaceData (RTLC6), so + # the update's ObjectMessage has no operation. It omits root, which is never removed from the + # pool (RTO5c2a) and still references "score", so the counter stays reachable from the root + # subscription. + mock_ws.send_to_client(build_object_sync_message('test', 'sync2:', [ + build_object_state('counter:score@1000', {'aaa': 't:1'}, counter={'count': 0}, + create_op={'counterCreate': {'count': 200}}), + ])) + await poll_until(lambda: len(events) >= 1, description='the sync-triggered event') + await settle() + + # Events from sync-triggered updates carry no message + for event in events: + assert event.message is None + + +# UTS: objects/unit/RTPO19f/subscribe-follows-path-0 +async def test_rtpo19f_subscribe_follows_path(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.get('score').subscribe(events.append) + + # Replace the counter at "score" with a new counter + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'score', {'objectId': 'counter:new@2000'}, remote_serial(0), 'remote'), + ])) + + # Increment the new counter at "score" + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:new@2000', 10, '100', 'remote'), + ])) + await poll_until(lambda: len(events) >= 2, description='the MAP_SET and COUNTER_INC events') + await settle() + + # The subscription follows the path, so it sees the new counter + found_new = False + for event in events: + if event.object.path() == 'score': + found_new = True + assert found_new is True + # The second dispatch is the increment on the new counter, which is what shows the + # subscription followed the path to the replacement rather than staying bound to the old + # object; the MAP_SET dispatch alone would satisfy found_new + assert len(events) == 2 + assert events[1].object.path() == 'score' + assert events[1].message.operation.action == ObjectOperationAction.COUNTER_INC + assert events[1].message.operation.object_id == 'counter:new@2000' + + +# UTS: objects/unit/RTPO19g/subscribe-no-side-effects-0 +async def test_rtpo19g_subscribe_no_side_effects(): + client, channel, root, mock_ws = await setup_synced_channel('test') + state_before = channel.state + # RTPO19g also rules out side effects on the RealtimeObject and on the channel itself, which + # the channel state alone does not show while it is attached + sync_state_before = channel.object._sync_state + sent_before = len(mock_ws.messages_from_client) + + root.get('score').subscribe(lambda event: None) + await settle() + + assert channel.state == state_before + assert channel.object._sync_state == sync_state_before + assert len(mock_ws.messages_from_client) == sent_before + + +# UTS: objects/unit/RTPO19/subscribe-primitive-path-0 +async def test_rtpo19_subscribe_primitive_path(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.get('name').subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the MAP_SET event at "name"') + await settle() + + assert len(events) == 1 + assert events[0].object.path() == 'name' + + +# UTS: objects/unit/RTPO19/map-clear-triggers-child-events-0 +async def test_rtpo19_map_clear_triggers_child_events(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.subscribe(events.append) + # UTS SPEC ERROR: the specification clears with serial "99", which sorts before the pool's + # entry serial "t:0", so RTLM24e1 removes no entry and the update carries no key; and it + # subscribes at the root only, which fires for any root update. Neither reaches the child + # paths the test is named for. The clear uses a serial after the pool's, and a subscription + # at the child path "name" is asserted as well. + child_events = [] + root.get('name').subscribe(child_events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_map_clear('root', remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the MAP_CLEAR event at the root') + + assert len(events) >= 1 + await poll_until(lambda: len(child_events) >= 1, description='the MAP_CLEAR event at "name"') + assert child_events[0].object.path() == 'name' + + +# UTS: objects/unit/RTPO19/child-events-bubble-0 +async def test_rtpo19_child_events_bubble(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.get('profile').subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('map:profile@1000', 'email', {'string': 'bob@example.com'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the profile event') + + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:nested@1000', 3, '100', 'remote'), + ])) + await poll_until(lambda: len(events) >= 2, description='the nested counter event') + + assert len(events) >= 2 + + +# UTS: objects/unit/RTO24c1/depth-filtering-formula-0 +async def test_rto24c1_depth_filtering_formula(): + client, channel, root, mock_ws = await setup_synced_channel('test') + # Seed a grandchild object under profile.prefs (path ["profile", "prefs", "deep"]), so the + # grandchild stimulus below can be a COUNTER_INC yielding only that depth-3 candidate (RTO6 + # creates counter:deep@3000 at its zero value) + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('map:prefs@1000', 'deep', {'objectId': 'counter:deep@3000'}, '50', 'remote'), + ])) + # The specification subscribes straight after the send, relying on the seed not reaching + # the listener under test; it is applied on the read task, and its ["profile", "prefs"] + # candidate is covered at depth 2, so it must land before the subscription + await poll_until(lambda: 'deep' in root.get('profile').get('prefs').as_live_map().keys(), + description='the seeding MAP_SET to be applied') + events = [] + # Subscribed at "profile" with depth 2: + # self (profile) -> ["profile"], 1 - 1 + 1 = 1 <= 2, covered + # child (profile.nested) -> ["profile", "nested_counter"], 2 - 1 + 1 = 2 <= 2, covered + # grandchild (prefs.deep) -> ["profile", "prefs", "deep"], 3 - 1 + 1 = 3 > 2, not covered + root.get('profile').subscribe(events.append, depth=2) + # Quiescence control: an unlimited-depth root listener covers the out-of-scope grandchild path + control = [] + root.subscribe(control.append) + + # Self event (profile map update): the first covered candidate is ["profile"] + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('map:profile@1000', 'email', {'string': 'bob@example.com'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the self event') + + # Child event (the nested counter, relative depth 2): covered + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:nested@1000', 3, '100', 'remote'), + ])) + await poll_until(lambda: len(events) >= 2, description='the child event') + await settle() + + # Grandchild event (counter:deep, relative depth 3): not covered. A COUNTER_INC yields only + # this candidate, where a MAP_SET on map:prefs would also yield the covered ["profile", "prefs"]. + control_before = len(control) + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:deep@3000', 1, '101', 'remote'), + ])) + await poll_until(lambda: len(control) > control_before, description='the control to see the grandchild event') + await settle() + + assert len(events) == 2 + + +# UTS: objects/unit/RTO24c1/prefix-mismatch-0 +async def test_rto24c1_prefix_mismatch(): + client, channel, root, mock_ws = await setup_synced_channel('test') + profile_events = [] + root.get('profile').subscribe(profile_events.append) + # Control listener at the root, which fires on both out-of-scope sends below + control_events = [] + root.subscribe(control_events.append) + + # Change at "score": "profile" is not a prefix of "score" + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 7, '99', 'remote'), + ])) + + # Change at "name": "profile" is not a prefix of "name" + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(control_events) >= 2, description='the control to see both events') + await settle() + + assert len(profile_events) == 0 + + +# UTS: objects/unit/RTO24b2a/candidate-paths-map-keys-0 +async def test_rto24b2a_candidate_paths_map_keys(): + client, channel, root, mock_ws = await setup_synced_channel('test') + score_events = [] + root_events = [] + # Subscribed at the child path ["score"] + root.get('score').subscribe(score_events.append) + # Subscribed at the root path [] + root.subscribe(root_events.append) + + # A MAP_SET on root with key "score" yields the candidates [] (root itself) and ["score"] + # (from the map update key), so both subscriptions fire + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'score', {'objectId': 'counter:new@2000'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(score_events) >= 1, description='the event at "score"') + await poll_until(lambda: len(root_events) >= 1, description='the event at the root') + await settle() + + assert len(score_events) == 1 + assert score_events[0].object.path() == 'score' + assert len(root_events) == 1 + + +# UTS: objects/unit/RTO24b2c/listener-exception-caught-0 +async def test_rto24b2c_listener_exception_caught(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + + def boom(event): + raise Exception('boom') + + root.subscribe(boom) + root.subscribe(events.append) + + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the second listener to fire') + await settle() + + assert len(events) == 1 + + +# UTS: objects/unit/RTO24b1/multi-path-dispatch-0 +async def test_rto24b1_multi_path_dispatch(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events_score = [] + events_alias = [] + + # "score" already points to counter:score@1000; a second reference "alias" gives it two paths + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'alias', {'objectId': 'counter:score@1000'}, '98', 'remote'), + ])) + # The specification subscribes straight after the send; the MAP_SET is applied on the read + # task, and its ["alias"] candidate would reach the "alias" subscription if it landed after it + await poll_until(lambda: 'alias' in root.keys(), description='the alias MAP_SET to be applied') + + root.get('score').subscribe(events_score.append) + root.get('alias').subscribe(events_alias.append) + + # Increment counter:score@1000, whose full paths are ["score"] and ["alias"] + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 5, '99', 'remote'), + ])) + await poll_until(lambda: len(events_score) >= 1, description='the event at "score"') + await poll_until(lambda: len(events_alias) >= 1, description='the event at "alias"') + await settle() + + assert len(events_score) == 1 + assert events_score[0].object.path() == 'score' + assert len(events_alias) == 1 + assert events_alias[0].object.path() == 'alias' + + +# UTS: objects/unit/RTO24b2b/fires-once-per-dispatch-0 +async def test_rto24b2b_fires_once_per_dispatch(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + # Subscribed at the root with unlimited depth, covering both [] and ["score"] + root.subscribe(events.append) + + # A MAP_SET on root with key "score" yields the candidates [] and ["score"]; the root + # subscription covers both and fires once, with the first + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'score', {'objectId': 'counter:new@2000'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(events) >= 1, description='the MAP_SET event') + + # Quiescence: a second, single-candidate dispatch is the control delivery, so a spurious + # second callback from the first dispatch would already have run + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:new@2000', 1, '100', 'remote'), + ])) + await poll_until(lambda: len(events) >= 2, description='the control COUNTER_INC event') + await settle() + + # One event per dispatch, though the first had two covered candidates + assert len(events) == 2 diff --git a/test/uts/objects/unit/path_object_test.py b/test/uts/objects/unit/path_object_test.py new file mode 100644 index 00000000..4d40bc2a --- /dev/null +++ b/test/uts/objects/unit/path_object_test.py @@ -0,0 +1,370 @@ +"""Derived from uts/objects/unit/path_object.md in ably/specification. + +Spec points: RTPO3, RTPO3a1, RTPO3b, RTPO3c1, RTPO4, RTPO4a, RTPO4b, RTPO4c, RTPO5, RTPO5b, RTPO5c, +RTPO5d, RTPO6, RTPO6b, RTPO6d, RTPO7, RTPO7a, RTPO7c, RTPO7d, RTPO7e, RTPO7f, RTPO8, RTPO8a, RTPO8c, +RTPO8f, RTPO9, RTPO9a, RTPO9c, RTPO9d, RTPO10, RTPO10a, RTPO10c, RTPO10d, RTPO11, RTPO11a, RTPO11c, +RTPO11d, RTPO12, RTPO12a, RTPO12c, RTPO12d, RTPO13, RTPO13a, RTPO13c1, RTPO13c2, RTPO13c3, RTPO13c4, +RTPO13c5, RTPO13d, RTPO14, RTPO14a, RTPO14b1, RTPO14b2, RTINS3b, RTINS4c, RTTS5d1 + +The specification reads through the untyped `PathObject`, which carries every method. ably-python +partitions it (LODR-061, RTTS3-RTTS6): navigation, `instance` and `compact_json` are on the base +`PathObject`, and the type-specific reads are reached through the unchecked view helpers, so +`po.value()` on a counter is `po.as_live_counter().value()`, on a primitive `po.as_primitive().value()`, +and `po.keys()` is `po.as_live_map().keys()`. A view whose type does not match what the path resolves +to answers None, or `[]` for the collection reads (RTTS5d1), which is what the specification's +null and empty-array expectations become. Where the specification expects the untyped `value()` to +be null, neither value view may answer, so both are asserted. + +The snapshot is `compact_json()` alone (RTTS3f), so the RTPO13 tests take it there and assert its +form where the two differ: binary as base64, and a map reached again as its object id (S-6). + +Every test opens with the standard synced channel; the module issues no writes, so no ACK is involved. +""" + +import pytest + +from ably.pubsub.liveobjects.instance import Instance +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import poll_until +from test.uts.objects.helpers.standard_test_pool import ( + build_map_set, + build_object_message, + setup_synced_channel, +) + + +async def _apply_back_ref(root, mock_ws): + """Sends the specification's `prefs.back_ref -> profile` MAP_SET, closing a cycle, and waits for it. + + The MAP_SET is an inbound OBJECT message, applied on the transport's read task, so the + specification's quiescence barrier is a poll until the new key is visible. + """ + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('map:prefs@1000', 'back_ref', {'objectId': 'map:profile@1000'}, '99', 'remote'), + ])) + await poll_until(lambda: 'back_ref' in root.get('profile').get('prefs').as_live_map().keys(), + description='the back_ref MAP_SET to be applied') + + +# UTS: objects/unit/RTPO4/path-string-representation-0 +async def test_rtpo4_path_string_representation(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + assert root.path() == '' + assert root.get('profile').path() == 'profile' + assert root.get('profile').get('email').path() == 'profile.email' + + +# UTS: objects/unit/RTPO4b/path-escapes-dots-0 +async def test_rtpo4b_path_escapes_dots(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + po = root.get('a.b').get('c') + + # The specification's "a\\.b.c" is the string a\.b.c: one backslash before the dot in the segment + assert po.path() == 'a\\.b.c' + + +# UTS: objects/unit/RTPO5/get-appends-key-0 +async def test_rtpo5_get_appends_key(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + child = root.get('profile') + grandchild = child.get('email') + + assert child.path() == 'profile' + assert grandchild.path() == 'profile.email' + assert child is not root + + +# UTS: objects/unit/RTPO5b/get-non-string-throws-0 +async def test_rtpo5b_get_non_string_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + with pytest.raises(AblyException) as excinfo: + root.get(123) + + assert excinfo.value.code == 40003 + + +# UTS: objects/unit/RTPO6/at-parses-path-0 +async def test_rtpo6_at_parses_path(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + po = root.at('profile.email') + + assert po.path() == 'profile.email' + assert po.as_primitive().value() == 'alice@example.com' + + +# UTS: objects/unit/RTPO6/at-escaped-dots-0 +async def test_rtpo6_at_escaped_dots(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + po = root.at('a\\.b.c') + + assert po.path() == 'a\\.b.c' + # UTS SPEC ERROR: the round trip through path() alone does not show that `\.` stayed inside + # its segment: an `at` that split on every dot gives the segments ['a\\', 'b', 'c'], which + # path() renders as the same string. RTPO6b makes `\.` a literal dot within a segment, so + # the parsed segments are asserted as well. + assert po._path == ['a.b', 'c'] + + +# UTS: objects/unit/RTPO6b/at-non-string-throws-0 +async def test_rtpo6b_at_non_string_throws(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + # NOTE: the features spec's RTPO6 defines no error for a path that is not a string (RTPO5b + # covers `get` only); 40003 is asserted as the specification has it, consistently with RTPO5b. + # ably-python's `at` also takes a sequence of segments (LODR-061), and an int is neither. + with pytest.raises(AblyException) as excinfo: + root.at(123) + + assert excinfo.value.code == 40003 + + +# UTS: objects/unit/RTPO7/value-counter-0 +async def test_rtpo7_value_counter(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + assert root.get('score').as_live_counter().value() == 100 + + +# UTS: objects/unit/RTPO7/value-primitive-0 +async def test_rtpo7_value_primitive(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + assert root.get('name').as_primitive().value() == 'Alice' + assert root.get('age').as_primitive().value() == 30 + assert root.get('active').as_primitive().value() is True + + +# UTS: objects/unit/RTPO7d/value-livemap-null-0 +async def test_rtpo7d_value_livemap_null(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + profile = root.get('profile') + + # value() on a map is null (RTPO7e); neither value view answers for a map (RTTS6b, RTTS6c) + assert profile.as_primitive().value() is None + assert profile.as_live_counter().value() is None + + +# UTS: objects/unit/RTPO7e/value-unresolvable-null-0 +async def test_rtpo7e_value_unresolvable_null(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + deep = root.get('nonexistent').get('deep') + + assert deep.as_primitive().value() is None + assert deep.as_live_counter().value() is None + + +# UTS: objects/unit/RTPO8/instance-live-object-0 +async def test_rtpo8_instance_live_object(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + counter_inst = root.get('score').instance() + assert isinstance(counter_inst, Instance) + assert counter_inst.id == 'counter:score@1000' + + map_inst = root.get('profile').instance() + assert isinstance(map_inst, Instance) + assert map_inst.id == 'map:profile@1000' + + +# UTS: objects/unit/RTPO8f/instance-primitive-wrapped-0 +async def test_rtpo8f_instance_primitive_wrapped(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + name_inst = root.get('name').instance() + + assert isinstance(name_inst, Instance) + assert name_inst.id is None + assert name_inst.as_primitive().value() == 'Alice' + + +# UTS: objects/unit/RTPO9/entries-yields-pairs-0 +async def test_rtpo9_entries_yields_pairs(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + entries = {} + for key, path_obj in root.entries(): + entries[key] = path_obj.path() + + assert entries['name'] == 'name' + assert entries['profile'] == 'profile' + assert len(entries) == 7 + + +# UTS: objects/unit/RTPO9d/entries-non-map-empty-0 +async def test_rtpo9d_entries_non_map_empty(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + entries = root.get('score').as_live_map().entries() + + assert len(entries) == 0 + + +# UTS: objects/unit/RTPO10/keys-returns-array-0 +async def test_rtpo10_keys_returns_array(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + keys = root.keys() + + assert isinstance(keys, list) + assert len(keys) == 7 + assert 'name' in keys + assert 'profile' in keys + assert 'score' in keys + + +# UTS: objects/unit/RTPO10d/keys-non-map-empty-0 +async def test_rtpo10d_keys_non_map_empty(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + keys = root.get('score').as_live_map().keys() + + assert isinstance(keys, list) + assert len(keys) == 0 + + +# UTS: objects/unit/RTPO11/values-returns-array-0 +async def test_rtpo11_values_returns_array(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + vals = root.values() + + assert isinstance(vals, list) + assert len(vals) == 7 + # Each element is a PathObject whose path is the key + paths = {} + for v in vals: + paths[v.path()] = True + assert paths['name'] is True + assert paths['profile'] is True + assert paths['score'] is True + + +# UTS: objects/unit/RTPO11d/values-non-map-empty-0 +async def test_rtpo11d_values_non_map_empty(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + vals = root.get('score').as_live_map().values() + + assert isinstance(vals, list) + assert len(vals) == 0 + + +# UTS: objects/unit/RTPO12/size-count-0 +async def test_rtpo12_size_count(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + assert root.size() == 7 + assert root.get('profile').as_live_map().size() == 3 + + +# UTS: objects/unit/RTPO12c/size-non-map-null-0 +async def test_rtpo12c_size_non_map_null(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + assert root.get('score').as_live_map().size() is None + assert root.get('name').as_live_map().size() is None + + +# UTS: objects/unit/RTPO13/compact-recursive-0 +async def test_rtpo13_compact_recursive(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + result = root.compact_json() + + assert result['name'] == 'Alice' + assert result['age'] == 30 + assert result['active'] is True + assert result['score'] == 100 + assert result['data'] == {'tags': ['a', 'b']} + # S-6: the specification's `compact()` keeps the bytes + assert result['avatar'] == 'AQID' + assert result['profile']['email'] == 'alice@example.com' + assert result['profile']['nested_counter'] == 5 + assert result['profile']['prefs']['theme'] == 'dark' + + +# UTS: objects/unit/RTPO13c5/compact-cycle-detection-0 +async def test_rtpo13c5_compact_cycle_detection(): + client, channel, root, mock_ws = await setup_synced_channel('test') + await _apply_back_ref(root, mock_ws) + + result = root.get('profile').compact_json() + + # S-6: the specification's `compact()` reuses the dict already built for the map + assert result['prefs']['back_ref'] == {'objectId': 'map:profile@1000'} + + +# UTS: objects/unit/RTPO13c/compact-counter-0 +async def test_rtpo13c_compact_counter(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + # S-6: the specification calls `compact()`, which reads the same for a counter + assert root.get('score').compact_json() == 100 + + +# UTS: objects/unit/RTPO14/compact-json-0 +async def test_rtpo14_compact_json(): + client, channel, root, mock_ws = await setup_synced_channel('test') + await _apply_back_ref(root, mock_ws) + + result = root.get('profile').compact_json() + + assert result['prefs']['back_ref'] == {'objectId': 'map:profile@1000'} + + +# UTS: objects/unit/RTPO3/path-resolution-walk-0 +async def test_rtpo3_path_resolution_walk(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + # The empty path resolves to the root map (RTPO3b), for which value() is null (RTPO7e) + assert root.as_primitive().value() is None + assert root.as_live_counter().value() is None + assert root.get('profile').get('prefs').get('theme').as_primitive().value() == 'dark' + + +# UTS: objects/unit/RTPO3a1/intermediate-not-map-0 +async def test_rtpo3a1_intermediate_not_map(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + something = root.get('score').get('something') + + assert something.as_primitive().value() is None + assert something.as_live_counter().value() is None + + +# UTS: objects/unit/RTPO3c1/read-null-on-failure-0 +async def test_rtpo3c1_read_null_on_failure(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + nonexistent = root.get('nonexistent') + + assert nonexistent.as_primitive().value() is None + assert nonexistent.as_live_counter().value() is None + assert nonexistent.instance() is None + assert nonexistent.as_live_map().size() is None + # S-6: the specification calls `compact()`, which reads the same where nothing resolves + assert nonexistent.compact_json() is None + + +# UTS: objects/unit/RTPO7/value-bytes-0 +async def test_rtpo7_value_bytes(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + assert root.get('avatar').as_primitive().value() == bytes([1, 2, 3]) + + +# UTS: objects/unit/RTPO14/compact-json-bytes-0 +async def test_rtpo14_compact_json_bytes(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + result = root.compact_json() + + assert result['avatar'] == 'AQID' diff --git a/test/uts/objects/unit/public_object_message_test.py b/test/uts/objects/unit/public_object_message_test.py new file mode 100644 index 00000000..f7930bbc --- /dev/null +++ b/test/uts/objects/unit/public_object_message_test.py @@ -0,0 +1,345 @@ +"""Derived from uts/objects/unit/public_object_message.md in ably/specification. + +Spec points: PAOM1, PAOM2, PAOM3, PAOOP1, PAOOP2, PAOOP3 + +The specification's `PublicObjectMessage.fromObjectMessage(source, channel)` is +`publicmessage.ObjectMessage._from_internal(source, channel_name)`, which takes the +channel's name rather than the channel (PAOM3b reads only `channel.name`), and +`PublicObjectOperation.fromObjectOperation(op)` is +`publicmessage.ObjectOperation._from_internal(op)`. The public types share their names with +the internal wire types, so the public pair is reached through the `publicmessage` module. + +Each source is built as the internal dataclasses directly. An attribute the specification +writes as `null` or omitted is None, operation actions are `ObjectOperationAction` members +and map semantics `ObjectsMapSemantics` members. + +The specification gives `mapCreateWithObjectId` and `counterCreateWithObjectId` the fields +of the payload they were derived from (`objectId`, `semantics`, `entries`, `count`), which +those payloads do not carry: MCRO2 and CCRO2 define only `initialValue` and `nonce`. Here +they carry those two, with `initialValue` the JSON encoding of the derived payload as +RTLMV4f and RTLCV4c produce it, and the derived payload in `derived_from` (RTLMV4j5, +RTLCV4g5). +""" + +import json + +from ably.pubsub.liveobjects import publicmessage +from ably.pubsub.liveobjects.objectmessage import ( + CounterCreate, + CounterCreateWithObjectId, + CounterInc, + MapClear, + MapCreate, + MapCreateWithObjectId, + MapRemove, + MapSet, + ObjectData, + ObjectDelete, + ObjectMessage, + ObjectOperation, + ObjectOperationAction, + ObjectsMapEntry, + ObjectsMapSemantics, +) + +NONCE = 'test-nonce-1234567890' + + +# UTS: objects/unit/PAOM3/construction-all-fields-0 +def test_paom3_construction_all_fields(): + source = ObjectMessage( + id='msg-id-1', + client_id='client-1', + connection_id='conn-1', + timestamp=1700000000000, + serial='01', + serial_timestamp=1700000001000, + site_code='site1', + extras={'key': 'value'}, + operation=ObjectOperation( + action=ObjectOperationAction.MAP_SET, + object_id='map:abc@1000', + map_set=MapSet(key='name', value=ObjectData(string='Alice')), + ), + ) + + public_msg = publicmessage.ObjectMessage._from_internal(source, 'test-channel') + + assert isinstance(public_msg, publicmessage.ObjectMessage) + assert public_msg.id == 'msg-id-1' + assert public_msg.client_id == 'client-1' + assert public_msg.connection_id == 'conn-1' + assert public_msg.timestamp == 1700000000000 + assert public_msg.channel == 'test-channel' + assert public_msg.serial == '01' + assert public_msg.serial_timestamp == 1700000001000 + assert public_msg.site_code == 'site1' + assert public_msg.extras == {'key': 'value'} + assert public_msg.operation is not None + assert isinstance(public_msg.operation, publicmessage.ObjectOperation) + assert public_msg.operation.action == ObjectOperationAction.MAP_SET + assert public_msg.operation.object_id == 'map:abc@1000' + assert public_msg.operation.map_set.key == 'name' + assert public_msg.operation.map_set.value.string == 'Alice' + + +# UTS: objects/unit/PAOM3/construction-optional-fields-missing-0 +def test_paom3_construction_optional_fields_missing(): + source = ObjectMessage( + operation=ObjectOperation( + action=ObjectOperationAction.COUNTER_INC, + object_id='counter:abc@1000', + counter_inc=CounterInc(number=5.0), + ), + ) + + public_msg = publicmessage.ObjectMessage._from_internal(source, 'my-channel') + + assert public_msg.id is None + assert public_msg.client_id is None + assert public_msg.connection_id is None + assert public_msg.timestamp is None + assert public_msg.channel == 'my-channel' + assert public_msg.serial is None + assert public_msg.serial_timestamp is None + assert public_msg.site_code is None + assert public_msg.extras is None + assert public_msg.operation is not None + assert public_msg.operation.action == ObjectOperationAction.COUNTER_INC + + +# UTS: objects/unit/PAOM3/channel-from-channel-name-0 +def test_paom3_channel_from_channel_name(): + source = ObjectMessage( + operation=ObjectOperation( + action=ObjectOperationAction.OBJECT_DELETE, + object_id='counter:abc@1000', + ), + ) + + public_msg = publicmessage.ObjectMessage._from_internal(source, 'different-channel-name') + + assert public_msg.channel == 'different-channel-name' + + +# UTS: objects/unit/PAOOP3/map-set-copies-fields-0 +def test_paoop3_map_set_copies_fields(): + source_operation = ObjectOperation( + action=ObjectOperationAction.MAP_SET, + object_id='map:abc@1000', + map_set=MapSet(key='color', value=ObjectData(string='blue')), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.MAP_SET + assert public_op.object_id == 'map:abc@1000' + assert public_op.map_set.key == 'color' + assert public_op.map_set.value.string == 'blue' + assert public_op.map_create is None + assert public_op.map_remove is None + assert public_op.counter_create is None + assert public_op.counter_inc is None + assert public_op.object_delete is None + assert public_op.map_clear is None + + +# UTS: objects/unit/PAOOP3/map-remove-copies-fields-0 +def test_paoop3_map_remove_copies_fields(): + source_operation = ObjectOperation( + action=ObjectOperationAction.MAP_REMOVE, + object_id='map:abc@1000', + map_remove=MapRemove(key='old-key'), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.MAP_REMOVE + assert public_op.object_id == 'map:abc@1000' + assert public_op.map_remove.key == 'old-key' + assert public_op.map_create is None + assert public_op.map_set is None + assert public_op.counter_create is None + assert public_op.counter_inc is None + assert public_op.object_delete is None + assert public_op.map_clear is None + + +# UTS: objects/unit/PAOOP3/counter-inc-copies-fields-0 +def test_paoop3_counter_inc_copies_fields(): + source_operation = ObjectOperation( + action=ObjectOperationAction.COUNTER_INC, + object_id='counter:abc@1000', + counter_inc=CounterInc(number=42.0), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.COUNTER_INC + assert public_op.object_id == 'counter:abc@1000' + assert public_op.counter_inc.number == 42 + assert public_op.map_create is None + assert public_op.map_set is None + assert public_op.map_remove is None + assert public_op.counter_create is None + assert public_op.object_delete is None + assert public_op.map_clear is None + + +# UTS: objects/unit/PAOOP3/object-delete-copies-fields-0 +def test_paoop3_object_delete_copies_fields(): + source_operation = ObjectOperation( + action=ObjectOperationAction.OBJECT_DELETE, + object_id='counter:abc@1000', + object_delete=ObjectDelete(), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.OBJECT_DELETE + assert public_op.object_id == 'counter:abc@1000' + assert public_op.object_delete is not None + assert public_op.map_create is None + assert public_op.map_set is None + assert public_op.map_remove is None + assert public_op.counter_create is None + assert public_op.counter_inc is None + assert public_op.map_clear is None + + +# UTS: objects/unit/PAOOP3/map-clear-copies-fields-0 +def test_paoop3_map_clear_copies_fields(): + source_operation = ObjectOperation( + action=ObjectOperationAction.MAP_CLEAR, + object_id='map:abc@1000', + map_clear=MapClear(), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.MAP_CLEAR + assert public_op.object_id == 'map:abc@1000' + assert public_op.map_clear is not None + assert public_op.map_create is None + assert public_op.map_set is None + assert public_op.map_remove is None + assert public_op.counter_create is None + assert public_op.counter_inc is None + assert public_op.object_delete is None + + +# UTS: objects/unit/PAOOP3/map-create-direct-0 +def test_paoop3_map_create_direct(): + source_operation = ObjectOperation( + action=ObjectOperationAction.MAP_CREATE, + object_id='map:new@2000', + map_create=MapCreate( + semantics=ObjectsMapSemantics.LWW, + entries={'key1': ObjectsMapEntry(data=ObjectData(string='val1'))}, + ), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.MAP_CREATE + assert public_op.object_id == 'map:new@2000' + assert public_op.map_create is not None + assert public_op.map_create.semantics == ObjectsMapSemantics.LWW + assert public_op.map_create.entries['key1'].data.string == 'val1' + assert public_op.counter_create is None + + +# UTS: objects/unit/PAOOP3/map-create-from-with-object-id-0 +def test_paoop3_map_create_from_with_object_id(): + derived_map_create = MapCreate( + semantics=ObjectsMapSemantics.LWW, + entries={'x': ObjectsMapEntry(data=ObjectData(number=10.0))}, + ) + + # The specification also lists `objectId`, `semantics` and `entries` inside + # `mapCreateWithObjectId`, which MCRO2 does not define; see the module docstring + source_operation = ObjectOperation( + action=ObjectOperationAction.MAP_CREATE, + object_id='map:derived@3000', + map_create_with_object_id=MapCreateWithObjectId( + initial_value=json.dumps(derived_map_create.to_dict()), + nonce=NONCE, + derived_from=derived_map_create, + ), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.MAP_CREATE + assert public_op.object_id == 'map:derived@3000' + assert public_op.map_create is not None + assert public_op.map_create.semantics == ObjectsMapSemantics.LWW + assert public_op.map_create.entries['x'].data.number == 10 + assert public_op.counter_create is None + # PAOOP1: the public operation carries the derived payload, never the WithObjectId one + assert public_op.map_create == derived_map_create + assert not hasattr(public_op, 'map_create_with_object_id') + + +# UTS: objects/unit/PAOOP3/counter-create-from-with-object-id-0 +def test_paoop3_counter_create_from_with_object_id(): + derived_counter_create = CounterCreate(count=100.0) + + # The specification also lists `objectId` and `count` inside + # `counterCreateWithObjectId`, which CCRO2 does not define; see the module docstring + source_operation = ObjectOperation( + action=ObjectOperationAction.COUNTER_CREATE, + object_id='counter:derived@3000', + counter_create_with_object_id=CounterCreateWithObjectId( + initial_value=json.dumps(derived_counter_create.to_dict()), + nonce=NONCE, + derived_from=derived_counter_create, + ), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.COUNTER_CREATE + assert public_op.object_id == 'counter:derived@3000' + assert public_op.counter_create is not None + assert public_op.counter_create.count == 100 + assert public_op.map_create is None + # PAOOP1: the public operation carries the derived payload, never the WithObjectId one + assert public_op.counter_create == derived_counter_create + assert not hasattr(public_op, 'counter_create_with_object_id') + + +# UTS: objects/unit/PAOOP3/create-payloads-omitted-0 +def test_paoop3_create_payloads_omitted(): + source_operation = ObjectOperation( + action=ObjectOperationAction.MAP_SET, + object_id='map:abc@1000', + map_set=MapSet(key='k', value=ObjectData(string='v')), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.map_create is None + assert public_op.counter_create is None + + +# UTS: objects/unit/PAOOP3/only-relevant-field-per-action-0 +def test_paoop3_only_relevant_field_per_action(): + source_operation = ObjectOperation( + action=ObjectOperationAction.COUNTER_CREATE, + object_id='counter:new@2000', + counter_create=CounterCreate(count=50.0), + ) + + public_op = publicmessage.ObjectOperation._from_internal(source_operation) + + assert public_op.action == ObjectOperationAction.COUNTER_CREATE + assert public_op.object_id == 'counter:new@2000' + assert public_op.counter_create is not None + assert public_op.counter_create.count == 50 + assert public_op.map_create is None + assert public_op.map_set is None + assert public_op.map_remove is None + assert public_op.counter_inc is None + assert public_op.object_delete is None + assert public_op.map_clear is None diff --git a/test/uts/objects/unit/realtime_object_test.py b/test/uts/objects/unit/realtime_object_test.py new file mode 100644 index 00000000..b230be97 --- /dev/null +++ b/test/uts/objects/unit/realtime_object_test.py @@ -0,0 +1,951 @@ +"""Derived from uts/objects/unit/realtime_object.md in ably/specification. + +Spec points: RTO2, RTO2a, RTO2b, RTO5c9, RTO10, RTO10a, RTO10b1, RTO10c1b, RTO10c1b1, RTO15, +RTO15e1, RTO15e2, RTO15e3, RTO17, RTO17b, RTO18, RTO18b, RTO18d, RTO18e, RTO19, RTO20, RTO20b, +RTO20c1, RTO20d1, RTO20d2, RTO20d4, RTO20e, RTO20e1, RTO20f, RTO23, RTO23a, RTO23c, RTO23c1, +RTO23d, RTO23e, RTO24a, RTO24c1, RTO25a, RTO25b, RTO26a, RTO26b, RTO26c, RTO27a, RTO27b, RTL33b, +RTL33c, RTLO4e10 + +The specification drives `channel.object` against the mock websocket, and reaches into the +RealtimeObject where nothing public observes what it asserts: the channel-state handler +(`_act_on_channel_state`), the pool (`_objects_pool`), the sync state (`_sync_state`) and the +GC interval (`_gc_interval_ms`). + +Several translations recur through the file. + +Every OBJECT the client publishes is ACKed, by the standard mock or by a test's own handler, +because a publish resolves only on its ACK (RTO15g). The one test whose specification sends +the ACK by hand, `ack-after-echo-no-double-apply`, drives the write as a task and ACKs it once +the OBJECT has left the client. + +The specifications' untyped `PathObject` calls go through the typed views: +`root.get("score").value()` is `root.get('score').as_live_counter().value()`, and +`root.get("name").value()` is `root.get('name').as_primitive().value()`. + +The modes a server grants travel as bits of the ATTACHED `flags`, so the specification's +`modes: ["OBJECT_SUBSCRIBE"]` is `flags=HAS_OBJECTS | OBJECT_SUBSCRIBE_FLAG`. + +An inbound frame is processed on the transport's read task, so state is read after a +`poll_until` or a `settle()`, and a negative assertion ("the listener did not fire", "the echo +did not apply") follows a positive control delivered behind the message under test. + +`enable_fake_timers()` is a `FakeClock` given to the client, which is where the GC timer +(RTO10a) is scheduled and where `now_ms()` is read for a sweep (RTO10c1b). +""" + +import asyncio + +import pytest + +from ably.pubsub.liveobjects.enums import ObjectsEvent, ObjectsSyncState +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction +from ably.pubsub.liveobjects.pathobject import PathObject +from ably.pubsub.liveobjects.valuetypes import LiveCounter +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import await_channel_state, poll_until +from test.uts.helpers.clock import FakeClock, settle +from test.uts.helpers.mock_websocket import MockWebSocket, channel_error_message, contains_in_order +from test.uts.objects.helpers.standard_test_pool import ( + ATTACH, + HAS_OBJECTS, + OBJECT, + OBJECT_PUBLISH_FLAG, + OBJECT_SUBSCRIBE_FLAG, + SITE_CODE, + STANDARD_POOL_OBJECTS, + ack_serial, + assert_unchanged_after_quiescence, + below_ack_serial, + build_ack_message, + build_counter_inc, + build_map_set, + build_object_delete, + build_object_message, + build_object_sync_message, + objects_attached_message, + objects_channel_options, + objects_client, + objects_connected_message, + remote_serial, + setup_synced_channel, + setup_synced_channel_no_ack, + standard_mock_websocket, +) + +# How long a test waits on an operation it expects to complete or fail, so that one which +# never settles fails with a timeout rather than running into the suite's own +OPERATION_TIMEOUT = 5 + +# RTO10b3's default grace period and RTO10a's example interval, which the GC tests advance past +DEFAULT_GC_GRACE_PERIOD_MS = 86_400_000 +DEFAULT_GC_INTERVAL_MS = 300_000 + + +def _score(root): + """The value of the standard pool's `score` counter, read through the counter view.""" + return root.get('score').as_live_counter().value() + + +def _nested_counter(root): + """The value of the standard pool's `profile.nested_counter` counter.""" + return root.get('profile').get('nested_counter').as_live_counter().value() + + +def _name(root): + """The standard pool's `name` primitive, read through the primitive view.""" + return root.get('name').as_primitive().value() + + +def _sent(mock_ws, action): + """How many protocol messages carrying `action` the client has sent.""" + return sum(1 for message in mock_ws.messages_from_client if message.get('action') == int(action)) + + +def _raise_if_failed(task): + """Re-raises the exception `task` finished with, so that an early failure reports itself.""" + if task.done() and not task.cancelled() and task.exception() is not None: + raise task.exception() + + +def _assert_pending(task, description): + """Asserts `task` is still waiting; a task that already failed re-raises its own error.""" + _raise_if_failed(task) + assert not task.done(), f'{description} completed while it should still be waiting' + + +async def _await_sent(mock_ws, action, task): + """Waits for the client to send a message carrying `action` on behalf of `task`.""" + await poll_until(lambda: task.done() or _sent(mock_ws, action) >= 1, + description=f'the client to send a message with action {int(action)}') + _raise_if_failed(task) + + +async def _restart_sync(channel, mock_ws, channel_serial='sync2:cursor'): + """Sends an ATTACHED carrying a sync cursor and waits for the sync state to leave SYNCED. + + A new ATTACHED moves the sync state to SYNCING (RTO4c), and the cursor keeps it there + until an OBJECT_SYNC completes the sequence. + """ + mock_ws.send_to_client(objects_attached_message(channel.name, channel_serial)) + await poll_until(lambda: channel.object._sync_state == ObjectsSyncState.SYNCING, + description='the ATTACHED to restart the objects sync') + + +def _acking_mock_websocket(serials, captured=None, **kwargs): + """The standard mock, ACKing each OBJECT with the serials `serials(message)` gives. + + Each OBJECT is recorded in `captured` first, if given. `kwargs` go to + `standard_mock_websocket`. + """ + mock_ws = None + + def on_object(message): + if captured is not None: + captured.append(message) + mock_ws.send_to_client(build_ack_message(message['msgSerial'], serials(message))) + + mock_ws = standard_mock_websocket(auto_ack=False, on_object=on_object, **kwargs) + return mock_ws + + +# UTS: objects/unit/RTO23/get-returns-path-object-0 +async def test_rto23_get_returns_path_object(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + assert isinstance(root, PathObject) + assert root._path == [] + # RTO23d: the root is the pool's InternalLiveMap with id `root` + assert root._root is channel.object._objects_pool['root'] + + +# UTS: objects/unit/RTO23a/get-requires-subscribe-mode-0 +async def test_rto23a_get_requires_subscribe_mode(): + mock_ws = standard_mock_websocket(sync_objects=None) + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options(ChannelMode.OBJECT_PUBLISH)) + + with pytest.raises(AblyException) as excinfo: + await channel.object.get() + + assert excinfo.value.code == 40024 + + +# UTS: objects/unit/RTO23e/get-reattaches-detached-0 +async def test_rto23e_get_reattaches_detached(): + client, channel, root, mock_ws = await setup_synced_channel('test', modes=(ChannelMode.OBJECT_SUBSCRIBE,)) + + await channel.detach() + assert channel.state == ChannelState.DETACHED + + root = await asyncio.wait_for(channel.object.get(), OPERATION_TIMEOUT) + + assert isinstance(root, PathObject) + assert root._path == [] + assert channel.state == ChannelState.ATTACHED + + +# UTS: objects/unit/RTO23c/get-waits-for-synced-0 +async def test_rto23c_get_waits_for_synced(): + mock_ws = standard_mock_websocket(attached_channel_serial='sync1:cursor', sync_objects=None) + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options()) + + task = asyncio.ensure_future(channel.object.get()) + await _await_sent(mock_ws, ProtocolMessageAction.ATTACH, task) + + # The ATTACHED carries a sync cursor and no OBJECT_SYNC has arrived, so get() must still + # be waiting for SYNCED + await settle() + _assert_pending(task, 'get()') + + mock_ws.send_to_client(build_object_sync_message('test', 'sync1:', STANDARD_POOL_OBJECTS)) + root = await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert isinstance(root, PathObject) + assert root._path == [] + + +# UTS: objects/unit/RTO23c1/fails-on-channel-detached-0 +async def test_rto23c1_fails_on_channel_detached(): + client, channel, root, mock_ws = await setup_synced_channel('test') + await _restart_sync(channel, mock_ws) + + task = asyncio.ensure_future(channel.object.get()) + await settle() + _assert_pending(task, 'get()') + + await channel.detach() + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert excinfo.value.code == 92008 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO23c1/fails-on-channel-suspended-0 +async def test_rto23c1_fails_on_channel_suspended(): + client, channel, root, mock_ws = await setup_synced_channel('test') + await _restart_sync(channel, mock_ws) + + task = asyncio.ensure_future(channel.object.get()) + await settle() + _assert_pending(task, 'get()') + + # The mock cannot drive the channel to SUSPENDED, so the RealtimeObject's channel-state + # handler is called directly, as the RTO27 test does + channel.object._act_on_channel_state(ChannelState.SUSPENDED) + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert excinfo.value.code == 92008 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO23c1/fails-on-channel-failed-0 +async def test_rto23c1_fails_on_channel_failed(): + client, channel, root, mock_ws = await setup_synced_channel('test') + await _restart_sync(channel, mock_ws) + + task = asyncio.ensure_future(channel.object.get()) + await settle() + _assert_pending(task, 'get()') + + mock_ws.send_to_client(channel_error_message('test', 90000, 'Channel failed', 400)) + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert excinfo.value.code == 92008 + assert excinfo.value.status_code == 400 + # The cause is the channel's errorReason, the injected channel error + assert excinfo.value.cause.code == 90000 + + +# UTS: objects/unit/RTO15/publish-sends-object-pm-0 +async def test_rto15_publish_sends_object_pm(): + captured = [] + mock_ws = _acking_mock_websocket(lambda message: ['serial-0'], captured) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + + await root.get('score').as_live_counter().increment(5) + + assert len(captured) == 1 + assert captured[0]['action'] == OBJECT + assert captured[0]['channel'] == 'test' + assert len(captured[0]['state']) == 1 + # RTO15e3: the state entry is the encoded ObjectMessage for the increment + operation = captured[0]['state'][0]['operation'] + assert operation['action'] == ObjectOperationAction.COUNTER_INC + assert operation['objectId'] == 'counter:score@1000' + assert operation['counterInc']['number'] == 5 + + +# UTS: objects/unit/RTO20/publish-and-apply-local-0 +async def test_rto20_publish_and_apply_local(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().increment(10) + + assert _score(root) == 110 + + +# UTS: objects/unit/RTO20c/missing-site-code-0 +async def test_rto20c_missing_site_code(): + captured = [] + mock_ws = _acking_mock_websocket(lambda message: ['serial-0'], captured, + connected=objects_connected_message(site_code=None)) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + + await root.get('score').as_live_counter().increment(10) + + # The increment was published and ACKed, so its not being applied is the siteCode's doing + assert len(captured) == 1 + assert _score(root) == 100 + + +# UTS: objects/unit/RTO20d1/null-serial-skipped-0 +async def test_rto20d1_null_serial_skipped(): + captured = [] + mock_ws = _acking_mock_websocket(lambda message: [None], captured) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + + await root.get('score').as_live_counter().increment(10) + + # The increment was published and ACKed, so its not being applied is the null serial's doing + assert len(captured) == 1 + assert _score(root) == 100 + + +# UTS: objects/unit/RTO20d4/empty-synthetic-list-skips-sync-wait-0 +async def test_rto20d4_empty_synthetic_list_skips_sync_wait(): + mock_ws = _acking_mock_websocket(lambda message: [None]) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + + # Back to SYNCING, where a write with something to apply would wait for SYNCED (RTO20e). + # No OBJECT_SYNC is ever sent, so the increment completing at all shows the wait was skipped. + await _restart_sync(channel, mock_ws) + + await asyncio.wait_for(root.get('score').as_live_counter().increment(10), OPERATION_TIMEOUT) + + assert channel.object._sync_state == ObjectsSyncState.SYNCING + assert _score(root) == 100 + + +# UTS: objects/unit/RTO20d4/mixed-null-serials-applies-non-null-0 +async def test_rto20d4_mixed_null_serials_applies_non_null(): + # The set() publishes [COUNTER_CREATE, MAP_SET] (RTLM20h1); the CREATE is ACKed with a + # null serial and skipped (RTO20d1), and the MAP_SET with a real one and applied + mock_ws = _acking_mock_websocket(lambda message: [None, ack_serial(message['msgSerial'], 1)]) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + + await root.set('child', LiveCounter.create(0)) + + assert 'child' in root.keys() + # The seven standard-pool entries and `child` + assert root.size() == 8 + + +# UTS: objects/unit/RTO20e/waits-for-synced-0 +async def test_rto20e_waits_for_synced(): + client, channel, root, mock_ws = await setup_synced_channel('test') + await _restart_sync(channel, mock_ws) + + task = asyncio.ensure_future(root.get('score').as_live_counter().increment(10)) + + # The OBJECT is published and ACKed; while still SYNCING the increment must wait, unapplied + await _await_sent(mock_ws, ProtocolMessageAction.OBJECT, task) + await settle() + _assert_pending(task, 'increment()') + assert _score(root) == 100 + + mock_ws.send_to_client(build_object_sync_message('test', 'sync2:', STANDARD_POOL_OBJECTS)) + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert _score(root) == 110 + + +# UTS: objects/unit/RTO20e1/fails-on-channel-detached-0 +async def test_rto20e1_fails_on_channel_detached(): + client, channel, root, mock_ws = await setup_synced_channel('test') + await _restart_sync(channel, mock_ws) + + task = asyncio.ensure_future(root.get('score').as_live_counter().increment(10)) + + # The publish and its ACK complete against the mock, and the increment waits for SYNCED + await _await_sent(mock_ws, ProtocolMessageAction.OBJECT, task) + await settle() + _assert_pending(task, 'increment()') + + await channel.detach() + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert excinfo.value.code == 92008 + # RTO20e1 gives the status code as well + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO20e1/fails-on-channel-failed-0 +async def test_rto20e1_fails_on_channel_failed(): + client, channel, root, mock_ws = await setup_synced_channel('test') + await _restart_sync(channel, mock_ws) + + task = asyncio.ensure_future(root.get('score').as_live_counter().increment(10)) + + # The publish and its ACK complete against the mock, and the increment waits for SYNCED + await _await_sent(mock_ws, ProtocolMessageAction.OBJECT, task) + await settle() + _assert_pending(task, 'increment()') + + mock_ws.send_to_client(channel_error_message('test', 90000, 'Channel failed', 400)) + + with pytest.raises(AblyException) as excinfo: + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert excinfo.value.code == 92008 + # RTO20e1 gives the status code, and the channel's errorReason as the cause + assert excinfo.value.status_code == 400 + assert excinfo.value.cause.code == 90000 + + +# UTS: objects/unit/RTO17/sync-state-events-0 +async def test_rto17_sync_state_events(): + mock_ws = standard_mock_websocket(attached_channel_serial='sync1:cursor', sync_objects=None) + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options()) + + events = [] + channel.object.on(ObjectsEvent.SYNCING, lambda: events.append('SYNCING')) + channel.object.on(ObjectsEvent.SYNCED, lambda: events.append('SYNCED')) + + task = asyncio.ensure_future(channel.object.get()) + await poll_until(lambda: task.done() or len(events) >= 1, description='the SYNCING event') + _raise_if_failed(task) + + mock_ws.send_to_client(build_object_sync_message('test', 'sync1:', STANDARD_POOL_OBJECTS)) + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert contains_in_order(events, ['SYNCING', 'SYNCED']), events + + +# UTS: objects/unit/RTO18d/duplicate-listener-0 +async def test_rto18d_duplicate_listener(): + client, channel, root, mock_ws = await setup_synced_channel('test') + calls = [] + + def listener(): + calls.append('SYNCED') + + channel.object.on(ObjectsEvent.SYNCED, listener) + channel.object.on(ObjectsEvent.SYNCED, listener) + + mock_ws.send_to_client(objects_attached_message('test', 'sync2:cursor')) + mock_ws.send_to_client(build_object_sync_message('test', 'sync2:', STANDARD_POOL_OBJECTS)) + await poll_until(lambda: len(calls) >= 2, description='the listener to be called twice') + await settle() + + assert len(calls) == 2 + + +# UTS: objects/unit/RTO19/off-deregisters-0 +async def test_rto19_off_deregisters(): + client, channel, root, mock_ws = await setup_synced_channel('test') + calls = [] + + def listener(): + calls.append('SYNCED') + + subscription = channel.object.on(ObjectsEvent.SYNCED, listener) + subscription.off() + + # A second listener on the same event is the control which shows the SYNCED was emitted + control = [] + channel.object.on(ObjectsEvent.SYNCED, lambda: control.append('SYNCED')) + + mock_ws.send_to_client(objects_attached_message('test', 'sync2:cursor')) + mock_ws.send_to_client(build_object_sync_message('test', 'sync2:', STANDARD_POOL_OBJECTS)) + await assert_unchanged_after_quiescence(lambda: len(calls), lambda: len(control) >= 1, + description='the control listener receives SYNCED') + + assert len(calls) == 0 + + +# UTS: objects/unit/RTO2/mode-enforcement-0 +async def test_rto2_mode_enforcement(): + # The server grants OBJECT_SUBSCRIBE only, where the channel requested both modes + mock_ws = standard_mock_websocket(attached_flags=HAS_OBJECTS | OBJECT_SUBSCRIBE_FLAG) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws) + + with pytest.raises(AblyException) as excinfo: + await root.set('name', 'Bob') + + assert excinfo.value.code == 40024 + + +# UTS: objects/unit/RTO23e/get-rejects-failed-0 +async def test_rto23e_get_rejects_failed(): + mock_ws = None + + def on_message_from_client(message): + if message.get('action') == ATTACH: + mock_ws.send_to_client(channel_error_message(message.get('channel'), 90000, 'Channel error', 400)) + + mock_ws = MockWebSocket( + on_connection_attempt=lambda conn: conn.respond_with_success(objects_connected_message()), + on_message_from_client=on_message_from_client, + ) + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options(ChannelMode.OBJECT_SUBSCRIBE)) + + # The attach fails, which leaves the channel FAILED + with pytest.raises(AblyException): + await channel.attach() + assert channel.state == ChannelState.FAILED + + with pytest.raises(AblyException) as excinfo: + await channel.object.get() + + assert excinfo.value.code == 90001 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO25a/access-requires-subscribe-mode-0 +async def test_rto25a_access_requires_subscribe_mode(): + mock_ws = standard_mock_websocket(attached_flags=HAS_OBJECTS | OBJECT_PUBLISH_FLAG) + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options(ChannelMode.OBJECT_PUBLISH)) + + with pytest.raises(AblyException) as excinfo: + await channel.object.get() + + assert excinfo.value.code == 40024 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO25b/access-throws-detached-0 +async def test_rto25b_access_throws_detached(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await channel.detach() + assert channel.state == ChannelState.DETACHED + + with pytest.raises(AblyException) as excinfo: + root.keys() + + assert excinfo.value.code == 90001 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO25b/access-throws-failed-0 +async def test_rto25b_access_throws_failed(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + mock_ws.send_to_client(channel_error_message('test', 90000, 'Channel error', 400)) + await await_channel_state(channel, ChannelState.FAILED) + + with pytest.raises(AblyException) as excinfo: + root.keys() + + assert excinfo.value.code == 90001 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO26a/write-requires-publish-mode-0 +async def test_rto26a_write_requires_publish_mode(): + mock_ws = standard_mock_websocket(attached_flags=HAS_OBJECTS | OBJECT_SUBSCRIBE_FLAG) + client, channel, root, mock_ws = await setup_synced_channel('test', mock_ws=mock_ws, + modes=(ChannelMode.OBJECT_SUBSCRIBE,)) + + with pytest.raises(AblyException) as excinfo: + await root.set('name', 'Bob') + + assert excinfo.value.code == 40024 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO26b/write-throws-detached-0 +async def test_rto26b_write_throws_detached(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await channel.detach() + assert channel.state == ChannelState.DETACHED + + with pytest.raises(AblyException) as excinfo: + await root.set('name', 'Bob') + + assert excinfo.value.code == 90001 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO26b/write-throws-failed-0 +async def test_rto26b_write_throws_failed(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + mock_ws.send_to_client(channel_error_message('test', 90000, 'Channel error', 400)) + await await_channel_state(channel, ChannelState.FAILED) + + with pytest.raises(AblyException) as excinfo: + await root.set('name', 'Bob') + + assert excinfo.value.code == 90001 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO26c/write-throws-echo-disabled-0 +async def test_rto26c_write_throws_echo_disabled(): + client, channel, root, mock_ws = await setup_synced_channel('test', echo_messages=False) + + with pytest.raises(AblyException) as excinfo: + await root.set('name', 'Bob') + + assert excinfo.value.code == 40000 + assert excinfo.value.status_code == 400 + + +# UTS: objects/unit/RTO24a/single-register-instance-0 +async def test_rto24a_single_register_instance(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events_root = [] + events_score = [] + + root.subscribe(events_root.append) + score_path = root.get('score') + score_path.subscribe(events_score.append) + + # `remote` has no entry in the counter's siteTimeserials, so the increment is new (RTLO4a) + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 5, 't:1', 'remote'), + ])) + await poll_until(lambda: len(events_score) >= 1, description='the score subscription to fire') + + # Both subscriptions are held by the one register, so both fire + assert len(events_root) >= 1 + assert len(events_score) >= 1 + + +# UTS: objects/unit/RTO24c1/coverage-prefix-depth-0 +async def test_rto24c1_coverage_prefix_depth(): + client, channel, root, mock_ws = await setup_synced_channel('test') + shallow_events = [] + deep_events = [] + + # With depth 1 a subscription at the root covers the root's own path only; a child such + # as ['score'] is at relative depth 2 (RTO24c2b) + root.subscribe(shallow_events.append, depth=1) + root.subscribe(deep_events.append) + + # A MAP_SET on the root itself, at path [] + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(0), 'remote'), + ])) + await poll_until(lambda: len(deep_events) >= 1, description='the deep subscription to see the root update') + + # An increment on the root's child, at path ['score'] + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 5, 't:2', 'remote'), + ])) + await poll_until(lambda: len(deep_events) >= 2, description='the deep subscription to see the child update') + await poll_until(lambda: len(shallow_events) >= 1, + description='the shallow subscription to see the root update') + await settle() + + # The shallow subscription sees the root update only; the deep one both + assert len(shallow_events) == 1 + assert len(deep_events) >= 2 + + +# UTS: objects/unit/RTO10/gc-tombstoned-objects-0 +async def test_rto10_gc_tombstoned_objects(): + clock = FakeClock() + client, channel, root, mock_ws = await setup_synced_channel('test', clock=clock) + pool = channel.object._objects_pool + + # A tombstone stamped now, which only the advance below makes eligible for collection + mock_ws.send_to_client(build_object_message('test', [ + build_object_delete('counter:score@1000', '99', 'site1', clock.now_ms()), + ])) + await poll_until(lambda: pool['counter:score@1000'].is_tombstone, + description='the OBJECT_DELETE to tombstone the score counter') + + await clock.advance(DEFAULT_GC_GRACE_PERIOD_MS + DEFAULT_GC_INTERVAL_MS) + + assert _score(root) is None + # UTS SPEC ERROR: the specification asserts only that `score` reads null, which holds + # from the moment the counter is tombstoned (RTLM14c, RTLM5d2h), with or without a GC + # sweep. The sweep itself is observed as the counter leaving the pool (RTO10c1b). + assert 'counter:score@1000' not in pool + + +# UTS: objects/unit/RTO10c1b1/gc-root-never-removed-0 +async def test_rto10c1b1_gc_root_never_removed(): + clock = FakeClock() + client, channel, root, mock_ws = await setup_synced_channel('test', clock=clock) + pool = channel.object._objects_pool + root_map = pool['root'] + + # A rogue OBJECT_DELETE targeting the root, which RTLO4e10 rejects + mock_ws.send_to_client(build_object_message('test', [ + build_object_delete('root', remote_serial(0), 'remote', clock.now_ms()), + ])) + # The score counter is tombstoned behind it: the control which shows the rogue delete has + # been processed, and, once collected, that a GC sweep has run past the grace period + mock_ws.send_to_client(build_object_message('test', [ + build_object_delete('counter:score@1000', '99', 'site1', clock.now_ms()), + ])) + await poll_until(lambda: pool['counter:score@1000'].is_tombstone, + description='the control OBJECT_DELETE to tombstone the score counter') + + # The root is not tombstoned and its data is untouched + assert _name(root) == 'Alice' + assert root_map.is_tombstone is False + + await clock.advance(DEFAULT_GC_GRACE_PERIOD_MS + DEFAULT_GC_INTERVAL_MS) + + assert 'counter:score@1000' not in pool + assert pool['root'] is root_map + + # The root is still live: an operation still applies to the root object the client holds + mock_ws.send_to_client(build_object_message('test', [ + build_map_set('root', 'name', {'string': 'Bob'}, remote_serial(1), 'remote'), + ])) + await poll_until(lambda: _name(root) == 'Bob', description='the MAP_SET to apply to the root') + + assert _name(root) == 'Bob' + + +# UTS: objects/unit/RTO20/echo-dedup-0 +async def test_rto20_echo_dedup(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().increment(10) + score_after_apply = _score(root) + + # The echo of the increment, carrying the serial it was ACKed and applied with + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 10, ack_serial(0, 0), SITE_CODE), + ])) + # A control behind the echo on the same connection, which shows the echo has been processed + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:nested@1000', 1, remote_serial(0), 'remote'), + ])) + await assert_unchanged_after_quiescence(lambda: _score(root), lambda: _nested_counter(root) == 6, + description='the control increment has applied') + score_after_echo = _score(root) + + assert score_after_apply == 110 + assert score_after_echo == 110 + + +# UTS: objects/unit/RTO20f/ack-no-site-timeserials-update-0 +async def test_rto20f_ack_no_site_timeserials_update(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().increment(10) + assert _score(root) == 110 + + # `below_ack_serial(9)` is not the ACK serial, so the echo dedup (RTO9a3) leaves it alone, + # and it sorts below `ack_serial(0, 0)`, so it is rejected as stale (RTLO4a) if and only if + # the LOCAL apply recorded the ACK serial in siteTimeserials + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 10, below_ack_serial(9), SITE_CODE), + ])) + await poll_until(lambda: _score(root) == 120, description='the inbound increment to apply') + + assert _score(root) == 120 + + +# UTS: objects/unit/RTO20/ack-after-echo-no-double-apply-0 +async def test_rto20_ack_after_echo_no_double_apply(): + client, channel, root, mock_ws = await setup_synced_channel_no_ack('test') + + task = asyncio.ensure_future(root.get('score').as_live_counter().increment(10)) + + # The OBJECT must be pending on the connection before its ACK can be matched to it + await _await_sent(mock_ws, ProtocolMessageAction.OBJECT, task) + + # The echo arrives before the ACK + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 10, ack_serial(0, 0), SITE_CODE), + ])) + mock_ws.send_to_client(build_ack_message(0, [ack_serial(0, 0)])) + + await asyncio.wait_for(task, OPERATION_TIMEOUT) + + assert _score(root) == 110 + + +# UTS: objects/unit/RTO5c9-RTO20/ack-serials-cleared-on-resync-0 +async def test_rto5c9_rto20_ack_serials_cleared_on_resync(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + await root.get('score').as_live_counter().increment(10) + assert _score(root) == 110 + + # A re-sync replaces the data and clears appliedOnAckSerials (RTO5c9) + mock_ws.send_to_client(objects_attached_message('test', 'sync2:cursor')) + mock_ws.send_to_client(build_object_sync_message('test', 'sync2:', STANDARD_POOL_OBJECTS)) + await poll_until(lambda: _score(root) == 100, description='the re-sync to replace the score') + assert _score(root) == 100 + + # The serial the increment was applied with on ACK applies normally once the set is cleared; + # were it still recorded, the dedup (RTO9a3) would discard it and the score would stay 100 + mock_ws.send_to_client(build_object_message('test', [ + build_counter_inc('counter:score@1000', 10, ack_serial(0, 0), SITE_CODE), + ])) + await poll_until(lambda: _score(root) == 110, description='the replayed increment to apply') + + assert _score(root) == 110 + + +# UTS: objects/unit/RTO20/subscription-fires-on-ack-apply-0 +async def test_rto20_subscription_fires_on_ack_apply(): + client, channel, root, mock_ws = await setup_synced_channel('test') + events = [] + root.get('score').subscribe(events.append) + + await root.get('score').as_live_counter().increment(10) + + assert len(events) >= 1 + assert _score(root) == 110 + + +# UTS: objects/unit/RTO23/get-implicit-attach-0 +async def test_rto23_get_implicit_attach(): + mock_ws = standard_mock_websocket() + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options()) + + assert channel.state == ChannelState.INITIALIZED + root = await asyncio.wait_for(channel.object.get(), OPERATION_TIMEOUT) + + assert isinstance(root, PathObject) + assert root._path == [] + assert channel.state == ChannelState.ATTACHED + + +# UTS: objects/unit/RTO23d/get-resolves-immediately-synced-0 +async def test_rto23d_get_resolves_immediately_synced(): + client, channel, root, mock_ws = await setup_synced_channel('test') + + root2 = await asyncio.wait_for(channel.object.get(), OPERATION_TIMEOUT) + + assert isinstance(root2, PathObject) + assert root2._path == [] + + +# UTS: objects/unit/RTO10b1/gc-grace-period-source-0 +async def test_rto10b1_gc_grace_period_source(): + clock = FakeClock() + mock_ws = standard_mock_websocket(connected=objects_connected_message(objects_gc_grace_period=5000)) + client = objects_client(mock_ws, clock=clock) + channel = client.channels.get('test', objects_channel_options()) + # UTS SPEC ERROR: the specification advances 6000 ms, which RTO10a's example interval of + # five minutes never sweeps within, so as written no GC check runs at all. The interval is + # shortened before the first ATTACHED schedules the GC timer, so that sweeps fall inside it. + channel.object._gc_interval_ms = 1000 + root = await channel.object.get() + pool = channel.object._objects_pool + + assert channel.object._gc_grace_period_ms == 5000 + + # A tombstone stamped now: eligible after 6000 ms under the 5000 ms grace period from + # ConnectionDetails, but not under the 24 hour default + mock_ws.send_to_client(build_object_message('test', [ + build_object_delete('counter:score@1000', '99', 'site1', clock.now_ms()), + ])) + await poll_until(lambda: pool['counter:score@1000'].is_tombstone, + description='the OBJECT_DELETE to tombstone the score counter') + + # The advance is taken in two steps so that the object is seen to survive the sweeps that + # fall inside the grace period (RTO10c1b) + await clock.advance(4000) + assert 'counter:score@1000' in pool + + await clock.advance(2000) + + assert _score(root) is None + # UTS SPEC ERROR: as in RTO10, `score` reads null from the moment the counter is + # tombstoned; the collection is observed as the counter leaving the pool + assert 'counter:score@1000' not in pool + + +# UTS: objects/unit/RTO17-RTO18/sync-event-sequences-0 +@pytest.mark.parametrize('scenario', [ + 'initial attach', + 're-sync on new ATTACHED', + 'ATTACHED without HAS_OBJECTS', +]) +async def test_rto17_rto18_sync_event_sequences(scenario): + if scenario == 'initial attach': + # A first attach is only observable on a channel that has not attached, with the + # listeners registered before attach() is called + mock_ws = standard_mock_websocket() + client = objects_client(mock_ws) + channel = client.channels.get('test', objects_channel_options()) + else: + client, channel, root, mock_ws = await setup_synced_channel('test') + + events = [] + channel.object.on(ObjectsEvent.SYNCING, lambda: events.append('SYNCING')) + channel.object.on(ObjectsEvent.SYNCED, lambda: events.append('SYNCED')) + + if scenario == 'initial attach': + await channel.attach() + elif scenario == 're-sync on new ATTACHED': + mock_ws.send_to_client(objects_attached_message('test', 'sync3:cursor')) + mock_ws.send_to_client(build_object_sync_message('test', 'sync3:', STANDARD_POOL_OBJECTS)) + else: + # RTO4c moves the sync state to SYNCING on any ATTACHED, and with no HAS_OBJECTS the + # sync completes at once (RTO4b4), moving it to SYNCED + mock_ws.send_to_client(objects_attached_message('test', 'sync4:', flags=0)) + + expected_events = ['SYNCING', 'SYNCED'] + await poll_until(lambda: len(events) >= len(expected_events), description=f'the {scenario} events') + await settle() + + assert events == expected_events + + +# UTS: objects/unit/RTO27/channel-state-data-lifecycle-0 +@pytest.mark.parametrize('state', [ChannelState.DETACHED, ChannelState.FAILED, ChannelState.SUSPENDED], + ids=['RTO27a-detached', 'RTO27a-failed', 'RTO27b-suspended']) +async def test_rto27_channel_state_data_lifecycle(state): + client, channel, root, mock_ws = await setup_synced_channel('test') + pool = channel.object._objects_pool + + # The standard pool has been materialised: the root map, a counter and a nested map + assert 'name' in pool['root'].data + assert pool['counter:score@1000'].value() == 100 + assert 'email' in pool['map:profile@1000'].data + + updates = [] + pool['root'].subscribe(updates.append) + pool['counter:score@1000'].subscribe(updates.append) + pool['map:profile@1000'].subscribe(updates.append) + + # The mock cannot drive every state, so the channel-state handler is called directly + channel.object._act_on_channel_state(state) + + if state == ChannelState.SUSPENDED: + # RTO27b: the data is retained unchanged + assert 'name' in pool['root'].data + assert pool['counter:score@1000'].value() == 100 + assert 'email' in pool['map:profile@1000'].data + else: + # RTO27a1: every object's data is cleared, the nested map's checked through the pool + # rather than through the root, while the objects themselves stay in the pool + assert pool['root'].data == {} + assert 'counter:score@1000' in pool + assert pool['counter:score@1000'].value() == 0 + assert 'map:profile@1000' in pool + assert pool['map:profile@1000'].data == {} + + # RTO27a1: and no update is emitted for the clear + await settle() + assert updates == [] diff --git a/test/uts/objects/unit/value_types_test.py b/test/uts/objects/unit/value_types_test.py new file mode 100644 index 00000000..0538c661 --- /dev/null +++ b/test/uts/objects/unit/value_types_test.py @@ -0,0 +1,291 @@ +"""Derived from uts/objects/unit/value_types.md in ably/specification. + +Spec points: RTLCV1, RTLCV2, RTLCV3, RTLCV3a1, RTLCV3b, RTLCV3c, RTLCV3d, RTLCV4, RTLCV4a, +RTLCV4b1, RTLCV4c, RTLCV4d, RTLCV4f, RTLCV4g1, RTLCV4g2, RTLCV4g3, RTLCV4g4, RTLCV4g5, +RTLMV1, RTLMV2, RTLMV3, RTLMV3a1, RTLMV3b, RTLMV3d, RTLMV4, RTLMV4a, RTLMV4b, RTLMV4c, +RTLMV4d, RTLMV4d1, RTLMV4d2, RTLMV4d3, RTLMV4d4, RTLMV4d5, RTLMV4d6, RTLMV4d7, RTLMV4e1, +RTLMV4e2, RTLMV4f, RTLMV4g, RTLMV4i, RTLMV4j1, RTLMV4j3, RTLMV4j4, RTLMV4j5, RTLMV4k + +Pure tests: no client and no mock. The blueprints' internal `count` and `entries` are +`_count` and `_entries`. The specification's `evaluate(vt)` is `evaluate(vt, timestamp_ms)`: +evaluation reads the server time (RTLCV4e, RTLMV4h), which the caller fetches and passes +in (S-3), so these tests pass a fixed one. + +Where the specification reads `operation.counterCreate` or `operation.mapCreate` of an +evaluated message, it means the create retained alongside the `*CreateWithObjectId` +payload (RTLCV4g5, RTLMV4j5). The evaluated operation carries only the +`*CreateWithObjectId` payload, and the retained create is read through +`resolved_counter_create` / `resolved_map_create`, its `derived_from` (S-4). Its entries +hold decoded values, so a `bytes` entry holds bytes; the base64 form is in the +`initialValue` JSON, which is encoded for the JSON wire (RTLMV4f1). +""" + +import json + +import pytest + +from ably.pubsub.liveobjects.objectmessage import ObjectOperationAction, ObjectsMapSemantics +from ably.pubsub.liveobjects.valuetypes import LiveCounter, LiveMap, evaluate +from ably.pubsub.util.exceptions import AblyException + +# The server time evaluation generates object ids with (S-3) +SERVER_TIME_MS = 1_700_000_000_000 + + +# UTS: objects/unit/RTLCV3/create-with-count-0 +def test_rtlcv3_create_with_count(): + vt = LiveCounter.create(42) + + assert isinstance(vt, LiveCounter) + assert vt._count == 42 + + +# UTS: objects/unit/RTLCV3/create-default-zero-0 +def test_rtlcv3_create_default_zero(): + vt = LiveCounter.create() + + assert vt._count == 0 + + +# UTS: objects/unit/RTLCV3c/no-validation-at-create-0 +def test_rtlcv3c_no_validation_at_create(): + # Does not raise + vt = LiveCounter.create('not_a_number') + + assert isinstance(vt, LiveCounter) + + +# UTS: objects/unit/RTLCV4/evaluate-generates-message-0 +def test_rtlcv4_evaluate_generates_message(): + vt = LiveCounter.create(42) + messages = evaluate(vt, SERVER_TIME_MS) + + assert len(messages) == 1 + msg = messages[0] + assert msg.operation.action == ObjectOperationAction.COUNTER_CREATE + assert msg.operation.object_id.startswith('counter:') + assert '@' in msg.operation.object_id + assert msg.operation.counter_create_with_object_id is not None + assert msg.operation.counter_create_with_object_id.nonce is not None + assert len(msg.operation.counter_create_with_object_id.nonce) >= 16 + assert msg.operation.counter_create_with_object_id.initial_value is not None + # RTLCV4c: the initial value is the JSON of the CounterCreate + assert json.loads(msg.operation.counter_create_with_object_id.initial_value) == {'count': 42} + + +# UTS: objects/unit/RTLCV4g5/retains-local-counter-create-0 +def test_rtlcv4g5_retains_local_counter_create(): + vt = LiveCounter.create(42) + messages = evaluate(vt, SERVER_TIME_MS) + + msg = messages[0] + # S-4: the specification's `operation.counterCreate` is the retained CounterCreate + assert msg.operation.resolved_counter_create is not None + assert msg.operation.resolved_counter_create.count == 42 + # The retained CounterCreate is local only and never sent (RTLCV4g5) + assert 'counterCreate' not in msg.operation.to_dict() + + +# UTS: objects/unit/RTLCV4a/evaluate-validates-count-0 +def test_rtlcv4a_evaluate_validates_count(): + vt = LiveCounter.create('not_a_number') + + with pytest.raises(AblyException) as excinfo: + evaluate(vt, SERVER_TIME_MS) + + assert excinfo.value.code == 40003 + + +# UTS: objects/unit/RTLCV4/evaluate-zero-count-0 +def test_rtlcv4_evaluate_zero_count(): + vt = LiveCounter.create(0) + messages = evaluate(vt, SERVER_TIME_MS) + + msg = messages[0] + # S-4: the retained CounterCreate + assert msg.operation.resolved_counter_create.count == 0 + + +# UTS: objects/unit/RTLMV3/create-with-entries-0 +def test_rtlmv3_create_with_entries(): + vt = LiveMap.create({ + 'name': 'Alice', + 'age': 30, + }) + + assert isinstance(vt, LiveMap) + assert vt._entries['name'] == 'Alice' + assert vt._entries['age'] == 30 + + +# UTS: objects/unit/RTLMV3/create-no-entries-0 +def test_rtlmv3_create_no_entries(): + vt = LiveMap.create() + + assert isinstance(vt, LiveMap) + + +# UTS: objects/unit/RTLMV4/evaluate-generates-message-0 +def test_rtlmv4_evaluate_generates_message(): + vt = LiveMap.create({'name': 'Alice'}) + messages = evaluate(vt, SERVER_TIME_MS) + + assert len(messages) == 1 + msg = messages[0] + assert msg.operation.action == ObjectOperationAction.MAP_CREATE + assert msg.operation.object_id.startswith('map:') + assert msg.operation.map_create_with_object_id is not None + assert len(msg.operation.map_create_with_object_id.nonce) >= 16 + assert msg.operation.map_create_with_object_id.initial_value is not None + # RTLMV4f: the initial value is the JSON of the encoded MapCreate + initial_value = json.loads(msg.operation.map_create_with_object_id.initial_value) + assert initial_value['semantics'] == ObjectsMapSemantics.LWW + assert initial_value['entries']['name']['data'] == {'string': 'Alice'} + + +# UTS: objects/unit/RTLMV4j5/retains-local-map-create-0 +def test_rtlmv4j5_retains_local_map_create(): + vt = LiveMap.create({'name': 'Alice'}) + messages = evaluate(vt, SERVER_TIME_MS) + + msg = messages[0] + # S-4: the specification's `operation.mapCreate` is the retained MapCreate + assert msg.operation.resolved_map_create is not None + assert msg.operation.resolved_map_create.semantics == ObjectsMapSemantics.LWW + assert msg.operation.resolved_map_create.entries['name'].data.string == 'Alice' + # The retained MapCreate is local only and never sent (RTLMV4j5) + assert 'mapCreate' not in msg.operation.to_dict() + + +# UTS: objects/unit/RTLMV4d/entry-value-types-0 +def test_rtlmv4d_entry_value_types(): + vt = LiveMap.create({ + 'str': 'hello', + 'num': 42, + 'bool': True, + 'json_arr': [1, 2, 3], + 'json_obj': {'key': 'value'}, + }) + messages = evaluate(vt, SERVER_TIME_MS) + + msg = messages[0] + # S-4: the retained MapCreate + entries = msg.operation.resolved_map_create.entries + assert entries['str'].data.string == 'hello' + assert entries['num'].data.number == 42 + # `bool` is an `int` in Python, so a boolean is told apart from a number by identity + assert entries['bool'].data.boolean is True + assert entries['json_arr'].data.json == [1, 2, 3] + assert entries['json_obj'].data.json == {'key': 'value'} + + +# UTS: objects/unit/RTLMV4d1/nested-value-types-0 +def test_rtlmv4d1_nested_value_types(): + inner_counter = LiveCounter.create(10) + inner_map = LiveMap.create({ + 'nested_count': inner_counter, + }) + outer = LiveMap.create({ + 'child': inner_map, + }) + messages = evaluate(outer, SERVER_TIME_MS) + + assert len(messages) == 3 + assert messages[0].operation.action == ObjectOperationAction.COUNTER_CREATE + assert messages[0].operation.object_id.startswith('counter:') + assert messages[1].operation.action == ObjectOperationAction.MAP_CREATE + assert messages[1].operation.object_id.startswith('map:') + assert messages[2].operation.action == ObjectOperationAction.MAP_CREATE + assert messages[2].operation.object_id.startswith('map:') + + inner_counter_id = messages[0].operation.object_id + inner_map_id = messages[1].operation.object_id + + # S-4: the retained MapCreates + assert messages[1].operation.resolved_map_create.entries['nested_count'].data.object_id == inner_counter_id + assert messages[2].operation.resolved_map_create.entries['child'].data.object_id == inner_map_id + + +# UTS: objects/unit/RTLMV4a/evaluate-validates-entries-0 +def test_rtlmv4a_evaluate_validates_entries(): + # The specification's input is `LiveMap.create(null)`. `LiveMap.create` takes None as its + # default, so `None` cannot be told apart from an omitted argument and the null input is + # not applicable. The other half of RTLMV4a, entries that are not a dict, is the + # reachable failure. + vt = LiveMap.create(['not', 'a', 'dict']) + + with pytest.raises(AblyException) as excinfo: + evaluate(vt, SERVER_TIME_MS) + + assert excinfo.value.code == 40003 + + # `None` is the omitted argument, and evaluates as an empty map (RTLMV4e2) + messages = evaluate(LiveMap.create(None), SERVER_TIME_MS) + assert messages[-1].operation.resolved_map_create.entries == {} + + +# UTS: objects/unit/RTLMV4b/evaluate-validates-keys-0 +def test_rtlmv4b_evaluate_validates_keys(): + vt = LiveMap.create({123: 'value'}) + + with pytest.raises(AblyException) as excinfo: + evaluate(vt, SERVER_TIME_MS) + + assert excinfo.value.code == 40003 + + +# UTS: objects/unit/RTLMV4c/evaluate-validates-values-0 +def test_rtlmv4c_evaluate_validates_values(): + vt = LiveMap.create({'fn': lambda: None}) + + with pytest.raises(AblyException) as excinfo: + evaluate(vt, SERVER_TIME_MS) + + assert excinfo.value.code == 40013 + + +# UTS: objects/unit/RTLMV4e2/empty-entries-0 +def test_rtlmv4e2_empty_entries(): + vt = LiveMap.create() + messages = evaluate(vt, SERVER_TIME_MS) + + msg = messages[0] + # S-4: the retained MapCreate + assert msg.operation.resolved_map_create.entries == {} + + +# The specification's `expected_value` for the bytes row, "AQID", is the JSON-wire form of +# bytes([1, 2, 3]). The retained MapCreate holds the decoded bytes, and the base64 is what the +# `initialValue` JSON carries, so the test asserts both. +MAP_SET_TYPE_SCENARIOS = [ + pytest.param('hello', 'string', 'hello', id='string'), + pytest.param(42, 'number', 42, id='number-42'), + pytest.param(3.14, 'number', 3.14, id='number-3.14'), + pytest.param(0, 'number', 0, id='number-0'), + pytest.param(-1, 'number', -1, id='number-minus-1'), + pytest.param(True, 'boolean', True, id='boolean-true'), + pytest.param(False, 'boolean', False, id='boolean-false'), + pytest.param([1, 'a', None], 'json', [1, 'a', None], id='json-array'), + pytest.param({'k': 'v'}, 'json', {'k': 'v'}, id='json-object'), + pytest.param(bytes([1, 2, 3]), 'bytes', bytes([1, 2, 3]), id='bytes'), +] + + +# UTS: objects/unit/RTLMV4d/map-set-all-types-table-0 +@pytest.mark.parametrize('value,expected_field,expected_value', MAP_SET_TYPE_SCENARIOS) +def test_rtlmv4d_map_set_all_types_table(value, expected_field, expected_value): + vt = LiveMap.create({'test_key': value}) + messages = evaluate(vt, SERVER_TIME_MS) + + # S-4: the retained MapCreate + entry = messages[0].operation.resolved_map_create.entries['test_key'] + actual = getattr(entry.data, expected_field) + if isinstance(expected_value, bool): + # `bool` is an `int` in Python, so a boolean is told apart from a number by identity + assert actual is expected_value + else: + assert actual == expected_value + + if expected_field == 'bytes': + initial_value = json.loads(messages[0].operation.map_create_with_object_id.initial_value) + assert initial_value['entries']['test_key']['data']['bytes'] == 'AQID' diff --git a/test/uts/realtime/integration/auth/token_renewal_test.py b/test/uts/realtime/integration/auth/token_renewal_test.py index a1c4b25f..e8e4407c 100644 --- a/test/uts/realtime/integration/auth/token_renewal_test.py +++ b/test/uts/realtime/integration/auth/token_renewal_test.py @@ -13,7 +13,7 @@ introduce a value no assertion consumes. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt diff --git a/test/uts/realtime/integration/auth/token_request_test.py b/test/uts/realtime/integration/auth/token_request_test.py index 285c8e26..9729b8c0 100644 --- a/test/uts/realtime/integration/auth/token_request_test.py +++ b/test/uts/realtime/integration/auth/token_request_test.py @@ -15,7 +15,7 @@ case, so `TokenParams(clientId: x)` is `{'client_id': x}`. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, sandbox_rest_client from test.uts.helpers.sandbox import random_id diff --git a/test/uts/realtime/integration/auth_test.py b/test/uts/realtime/integration/auth_test.py index 7f82d260..3bb60ae2 100644 --- a/test/uts/realtime/integration/auth_test.py +++ b/test/uts/realtime/integration/auth_test.py @@ -15,7 +15,7 @@ registered beforehand by the time the call completes. The reauth test needs no settle. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client from test.uts.helpers.deviations import deviation from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt, random_id diff --git a/test/uts/realtime/integration/channel_history_test.py b/test/uts/realtime/integration/channel_history_test.py index d49eb95a..ae16404e 100644 --- a/test/uts/realtime/integration/channel_history_test.py +++ b/test/uts/realtime/integration/channel_history_test.py @@ -19,7 +19,7 @@ specification writes raises `ValueError`. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until from test.uts.helpers.sandbox import random_id diff --git a/test/uts/realtime/integration/channels/channel_attach_test.py b/test/uts/realtime/integration/channels/channel_attach_test.py index 68f31243..707ffb5d 100644 --- a/test/uts/realtime/integration/channels/channel_attach_test.py +++ b/test/uts/realtime/integration/channels/channel_attach_test.py @@ -6,7 +6,7 @@ `use_binary_protocol`; `sandbox_realtime_client` already defaults to it. `RealtimeChannel.publish()` takes its arguments positionally. The keyword form the -specification writes, which `RestChannel.publish()` does accept, raises +specification writes, which `HttpChannel.publish()` does accept, raises `ValueError: publish() expects either (name, data) or a message object or array of messages` before anything reaches the server. @@ -18,9 +18,9 @@ import pytest -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import await_connection_state, sandbox_realtime_client from test.uts.helpers.sandbox import random_id diff --git a/test/uts/realtime/integration/channels/channel_publish_test.py b/test/uts/realtime/integration/channels/channel_publish_test.py index 53b9800a..d106a243 100644 --- a/test/uts/realtime/integration/channels/channel_publish_test.py +++ b/test/uts/realtime/integration/channels/channel_publish_test.py @@ -6,7 +6,7 @@ per protocol and passes `use_binary_protocol` to both clients it builds. `RealtimeChannel.publish()` takes its arguments positionally; the keyword form the -specification writes, which `RestChannel.publish()` does accept, raises `ValueError`. +specification writes, which `HttpChannel.publish()` does accept, raises `ValueError`. A binary payload arrives as a `bytearray` rather than `bytes` under either protocol, so the type assertion reads both. `bytearray` compares equal to the `bytes` that was @@ -22,8 +22,8 @@ spells out for the same wait. """ -from ably.realtime.connection import ConnectionState -from ably.types.message import Message +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.message import Message from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until from test.uts.helpers.sandbox import random_id diff --git a/test/uts/realtime/integration/channels/channel_subscribe_test.py b/test/uts/realtime/integration/channels/channel_subscribe_test.py index c8850930..99a1022e 100644 --- a/test/uts/realtime/integration/channels/channel_subscribe_test.py +++ b/test/uts/realtime/integration/channels/channel_subscribe_test.py @@ -20,7 +20,7 @@ spells out for the same wait. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until from test.uts.helpers.sandbox import random_id diff --git a/test/uts/realtime/integration/connection/connection_failures_test.py b/test/uts/realtime/integration/connection/connection_failures_test.py index 3e31a24e..fa4cf3b9 100644 --- a/test/uts/realtime/integration/connection/connection_failures_test.py +++ b/test/uts/realtime/integration/connection/connection_failures_test.py @@ -18,7 +18,7 @@ 1008. It is noise from teardown, not a failure of either test. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client # The wait the specification gives each connect attempt. diff --git a/test/uts/realtime/integration/connection_lifecycle_test.py b/test/uts/realtime/integration/connection_lifecycle_test.py index ffa0c126..d58a66ba 100644 --- a/test/uts/realtime/integration/connection_lifecycle_test.py +++ b/test/uts/realtime/integration/connection_lifecycle_test.py @@ -16,7 +16,7 @@ import re -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client # The waits the specification's `Integration Test Notes` give each leg: auth plus transport diff --git a/test/uts/realtime/integration/delta_decoding_test.py b/test/uts/realtime/integration/delta_decoding_test.py index 6538d84b..f6c8ef48 100644 --- a/test/uts/realtime/integration/delta_decoding_test.py +++ b/test/uts/realtime/integration/delta_decoding_test.py @@ -7,7 +7,7 @@ vcdiff delta, and the SDK decodes it against the payload it stored for the previous one. The specification's `plugins: { vcdiff: decoder }` is the `vcdiff_decoder` client option -here, and its `VCDiffDecoder` interface is `ably.types.options.VCDiffDecoder` — +here, and its `VCDiffDecoder` interface is `ably.pubsub.types.options.VCDiffDecoder` — `decode(delta, base) -> bytes`, the VD2a argument order. `AblyVCDiffDecoder` is the real implementation, backed by the `vcdiff-decoder` library; `CountingDecoder` wraps it so a test can assert how many deltas the server actually sent. @@ -20,11 +20,11 @@ import os -from ably import AblyVCDiffDecoder -from ably.realtime.connection import ConnectionState -from ably.types.channeloptions import ChannelOptions -from ably.types.channelstate import ChannelState -from ably.types.options import VCDiffDecoder +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server import AblyVCDiffDecoder +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.options import VCDiffDecoder from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/integration/mutable_messages_test.py b/test/uts/realtime/integration/mutable_messages_test.py index 8d19691a..481f50da 100644 --- a/test/uts/realtime/integration/mutable_messages_test.py +++ b/test/uts/realtime/integration/mutable_messages_test.py @@ -18,15 +18,15 @@ consistent; the two polls below are the specification's `poll_until_success` around them. """ -from ably.http.paginatedresult import PaginatedResult -from ably.realtime.connection import ConnectionState -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.channelmode import ChannelMode -from ably.types.channeloptions import ChannelOptions -from ably.types.channelstate import ChannelState -from ably.types.message import Message, MessageAction -from ably.types.operations import MessageOperation, UpdateDeleteResult -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.message import Message, MessageAction +from ably.pubsub.types.operations import MessageOperation, UpdateDeleteResult +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/integration/presence/presence_sync_test.py b/test/uts/realtime/integration/presence/presence_sync_test.py index 6a78481d..20a46ed6 100644 --- a/test/uts/realtime/integration/presence/presence_sync_test.py +++ b/test/uts/realtime/integration/presence/presence_sync_test.py @@ -17,8 +17,8 @@ behalf of others is built with `client_id='*'`. """ -from ably.realtime.connection import ConnectionState -from ably.types.presence import PresenceAction +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.presence import PresenceAction from test.uts.helpers.client import await_connection_state, sandbox_realtime_client from test.uts.helpers.sandbox import random_id diff --git a/test/uts/realtime/integration/presence_lifecycle_test.py b/test/uts/realtime/integration/presence_lifecycle_test.py index a6a74603..0a8a7c70 100644 --- a/test/uts/realtime/integration/presence_lifecycle_test.py +++ b/test/uts/realtime/integration/presence_lifecycle_test.py @@ -22,8 +22,8 @@ import asyncio -from ably.realtime.connection import ConnectionState -from ably.types.presence import PresenceAction +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.presence import PresenceAction from test.uts.helpers.client import ( await_connection_state, sandbox_realtime_client, diff --git a/test/uts/realtime/integration/proxy/auth_reauth_test.py b/test/uts/realtime/integration/proxy/auth_reauth_test.py index 847f66f1..8e47f75e 100644 --- a/test/uts/realtime/integration/proxy/auth_reauth_test.py +++ b/test/uts/realtime/integration/proxy/auth_reauth_test.py @@ -22,7 +22,7 @@ an `auth` attribute left the client, and that the connection was never disturbed. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt diff --git a/test/uts/realtime/integration/proxy/channel_faults_test.py b/test/uts/realtime/integration/proxy/channel_faults_test.py index dbcdd378..09ae9907 100644 --- a/test/uts/realtime/integration/proxy/channel_faults_test.py +++ b/test/uts/realtime/integration/proxy/channel_faults_test.py @@ -28,9 +28,9 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/integration/proxy/connection_open_failures_test.py b/test/uts/realtime/integration/proxy/connection_open_failures_test.py index 58206871..9b7de332 100644 --- a/test/uts/realtime/integration/proxy/connection_open_failures_test.py +++ b/test/uts/realtime/integration/proxy/connection_open_failures_test.py @@ -26,8 +26,8 @@ callback, because renewing a token is what it is about. """ -from ably import AblyRest -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server import create_http_client from test.uts.helpers.client import await_connection_state, sandbox_realtime_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_websocket import contains_in_order @@ -99,7 +99,7 @@ def token_auth_callback(api_key): again; the token arrives over a connection the proxy never sees. """ async def auth_callback(params): - inner_rest = AblyRest(key=api_key, endpoint=SANDBOX_ENDPOINT) + inner_rest = create_http_client(key=api_key, endpoint=SANDBOX_ENDPOINT) try: return await inner_rest.auth.request_token() finally: diff --git a/test/uts/realtime/integration/proxy/connection_resume_test.py b/test/uts/realtime/integration/proxy/connection_resume_test.py index 22e042b9..6f93dff8 100644 --- a/test/uts/realtime/integration/proxy/connection_resume_test.py +++ b/test/uts/realtime/integration/proxy/connection_resume_test.py @@ -39,8 +39,8 @@ import asyncio import json -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import sandbox_realtime_client, sandbox_rest_client, wall_clock_poll_until from test.uts.helpers.deviations import deviation from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt, random_id diff --git a/test/uts/realtime/integration/proxy/heartbeat_test.py b/test/uts/realtime/integration/proxy/heartbeat_test.py index b9602be6..c90a1b90 100644 --- a/test/uts/realtime/integration/proxy/heartbeat_test.py +++ b/test/uts/realtime/integration/proxy/heartbeat_test.py @@ -27,7 +27,7 @@ waiting for it, so the reconnection settles instead of cycling. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, sandbox_realtime_client, wall_clock_poll_until from test.uts.helpers.mock_websocket import contains_in_order from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt diff --git a/test/uts/realtime/integration/proxy/presence_reentry_test.py b/test/uts/realtime/integration/proxy/presence_reentry_test.py index 9e20a4c0..674ae6e9 100644 --- a/test/uts/realtime/integration/proxy/presence_reentry_test.py +++ b/test/uts/realtime/integration/proxy/presence_reentry_test.py @@ -26,8 +26,8 @@ from datetime import datetime -from ably.realtime.connection import ConnectionState -from ably.types.channelstate import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.channelstate import ChannelState from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/integration/proxy/rest_faults_test.py b/test/uts/realtime/integration/proxy/rest_faults_test.py index fa8d2c1a..dcbf0f38 100644 --- a/test/uts/realtime/integration/proxy/rest_faults_test.py +++ b/test/uts/realtime/integration/proxy/rest_faults_test.py @@ -25,10 +25,10 @@ import pytest -from ably import AblyRest -from ably.realtime.connection import ConnectionState -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.server import create_http_client +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, @@ -67,7 +67,7 @@ def token_auth_callback(api_key, invocations=None): async def auth_callback(params): if invocations is not None: invocations.append(params) - inner_rest = AblyRest(key=api_key, endpoint=SANDBOX_ENDPOINT) + inner_rest = create_http_client(key=api_key, endpoint=SANDBOX_ENDPOINT) try: return await inner_rest.auth.request_token() finally: diff --git a/test/uts/realtime/unit/auth/auth_callback_errors_test.py b/test/uts/realtime/unit/auth/auth_callback_errors_test.py index 6b941e92..7f056457 100644 --- a/test/uts/realtime/unit/auth/auth_callback_errors_test.py +++ b/test/uts/realtime/unit/auth/auth_callback_errors_test.py @@ -8,10 +8,10 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import await_connection_state, poll_until, realtime_client, rest_client from test.uts.helpers.clock import FakeClock, settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/auth/connection_auth_test.py b/test/uts/realtime/unit/auth/connection_auth_test.py index 8beaa42f..a1c32b77 100644 --- a/test/uts/realtime/unit/auth/connection_auth_test.py +++ b/test/uts/realtime/unit/auth/connection_auth_test.py @@ -5,10 +5,10 @@ import time -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import await_connection_state, poll_until, realtime_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_websocket import MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/auth/realtime_authorize_test.py b/test/uts/realtime/unit/auth/realtime_authorize_test.py index 568611c0..aa9589bf 100644 --- a/test/uts/realtime/unit/auth/realtime_authorize_test.py +++ b/test/uts/realtime/unit/auth/realtime_authorize_test.py @@ -8,11 +8,11 @@ import pytest -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionEvent, ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionEvent, ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/auth/token_expiry_non_renewable_test.py b/test/uts/realtime/unit/auth/token_expiry_non_renewable_test.py index 2c36c07e..c8dbaab6 100644 --- a/test/uts/realtime/unit/auth/token_expiry_non_renewable_test.py +++ b/test/uts/realtime/unit/auth/token_expiry_non_renewable_test.py @@ -5,7 +5,7 @@ import logging -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_websocket import ERROR_MESSAGE, MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/channels/channel_additional_attached_test.py b/test/uts/realtime/unit/channels/channel_additional_attached_test.py index eb9b64e9..3cd559f7 100644 --- a/test/uts/realtime/unit/channels/channel_additional_attached_test.py +++ b/test/uts/realtime/unit/channels/channel_additional_attached_test.py @@ -3,10 +3,10 @@ Spec points: RTL12 """ -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.mock_websocket import MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/channels/channel_annotations_test.py b/test/uts/realtime/unit/channels/channel_annotations_test.py index 61fa13f5..b70a1e4a 100644 --- a/test/uts/realtime/unit/channels/channel_annotations_test.py +++ b/test/uts/realtime/unit/channels/channel_annotations_test.py @@ -21,13 +21,13 @@ import pytest -from ably.realtime.annotations import RealtimeAnnotations -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.annotations import RealtimeAnnotations +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import await_connection_state, poll_until, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.mock_websocket import MockWebSocket, attached_message, connected_message @@ -378,7 +378,7 @@ async def test_rtan4e_subscribe_warns_no_mode(caplog): mock_ws = annotating_mock(channel_name, flags=Flag.PUBLISH) channel = await attached_channel(mock_ws, channel_name) - with caplog.at_level('WARNING', logger='ably.realtime.annotations'): + with caplog.at_level('WARNING', logger='ably.pubsub.realtime.annotations'): await channel.annotations.subscribe(lambda annotation: None) warnings = [record.getMessage() for record in caplog.records if record.levelname == 'WARNING'] @@ -402,7 +402,7 @@ async def test_rtan4e1_no_warn_unattached(caplog): assert channel.state == ChannelState.INITIALIZED - with caplog.at_level('WARNING', logger='ably.realtime.annotations'): + with caplog.at_level('WARNING', logger='ably.pubsub.realtime.annotations'): subscribing = asyncio.ensure_future(channel.annotations.subscribe(lambda annotation: None)) await poll_until(lambda: channel.state == ChannelState.ATTACHING, description='the implicit attach is in flight') diff --git a/test/uts/realtime/unit/channels/channel_attach_test.py b/test/uts/realtime/unit/channels/channel_attach_test.py index 1be3771c..8f6b05d7 100644 --- a/test/uts/realtime/unit/channels/channel_attach_test.py +++ b/test/uts/realtime/unit/channels/channel_attach_test.py @@ -8,13 +8,13 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelmode import ChannelMode -from ably.types.channeloptions import ChannelOptions -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/channels/channel_attributes_test.py b/test/uts/realtime/unit/channels/channel_attributes_test.py index 78d09e00..aeeeaff2 100644 --- a/test/uts/realtime/unit/channels/channel_attributes_test.py +++ b/test/uts/realtime/unit/channels/channel_attributes_test.py @@ -7,10 +7,10 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/channels/channel_connection_state_test.py b/test/uts/realtime/unit/channels/channel_connection_state_test.py index b2b65f50..d1b2a947 100644 --- a/test/uts/realtime/unit/channels/channel_connection_state_test.py +++ b/test/uts/realtime/unit/channels/channel_connection_state_test.py @@ -7,10 +7,10 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/channels/channel_delta_decoding_test.py b/test/uts/realtime/unit/channels/channel_delta_decoding_test.py index fa71c466..c3a3262e 100644 --- a/test/uts/realtime/unit/channels/channel_delta_decoding_test.py +++ b/test/uts/realtime/unit/channels/channel_delta_decoding_test.py @@ -22,10 +22,10 @@ import base64 import uuid -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.types.options import VCDiffDecoder +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.options import VCDiffDecoder from test.uts.helpers.client import await_connection_state, poll_until, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/channels/channel_detach_test.py b/test/uts/realtime/unit/channels/channel_detach_test.py index 90c8b397..e5a2a17d 100644 --- a/test/uts/realtime/unit/channels/channel_detach_test.py +++ b/test/uts/realtime/unit/channels/channel_detach_test.py @@ -7,10 +7,10 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/channels/channel_error_test.py b/test/uts/realtime/unit/channels/channel_error_test.py index b135966b..268eca60 100644 --- a/test/uts/realtime/unit/channels/channel_error_test.py +++ b/test/uts/realtime/unit/channels/channel_error_test.py @@ -7,10 +7,10 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/channels/channel_get_message_test.py b/test/uts/realtime/unit/channels/channel_get_message_test.py index 7a2cb2ab..4019ea2a 100644 --- a/test/uts/realtime/unit/channels/channel_get_message_test.py +++ b/test/uts/realtime/unit/channels/channel_get_message_test.py @@ -4,13 +4,13 @@ The specification points at uts/rest/unit/channel/get_message.md (RSL11) rather than listing steps of its own, so its single Test ID becomes one test driving an -`AblyRealtime` over the HTTP mock and asserting the core observable of the derived +`DefaultPubSubRealtimeClient` over the HTTP mock and asserting the core observable of the derived REST suite: the endpoint the call reaches and the `Message` it returns. """ import uuid -from ably.types.message import Message +from ably.pubsub.types.message import Message from test.uts.helpers.client import realtime_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/realtime/unit/channels/channel_history_test.py b/test/uts/realtime/unit/channels/channel_history_test.py index 8f5647b2..3675e239 100644 --- a/test/uts/realtime/unit/channels/channel_history_test.py +++ b/test/uts/realtime/unit/channels/channel_history_test.py @@ -3,7 +3,7 @@ Spec points: RTL10, RTL10a, RTL10b, RTL10c RTL10a points at uts/rest/unit/channel/history.md (RSL2) rather than listing steps of -its own, so its Test ID becomes one test driving an `AblyRealtime` over the HTTP mock +its own, so its Test ID becomes one test driving an `DefaultPubSubRealtimeClient` over the HTTP mock and asserting the core observable of the derived REST suite. The two RTL10b tests cover `untilAttach`, which ably-python does not implement; see @@ -14,11 +14,11 @@ import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient @@ -59,9 +59,9 @@ def on_message_from_client(msg): # UTS: realtime/unit/RTL10a/supports-rest-params-0 async def test_rtl10a_supports_rest_params(): # The specification directs uts/rest/unit/channel/history.md (RSL2) at a realtime - # channel in place of a REST one. `AblyRealtime` subclasses `AblyRest`, so the same - # HTTP mock serves it, and this mirrors that suite's `RSL2a/returns-paginated-result-0` - # and `RSL2b/query-parameters-0`. + # channel in place of a REST one. `DefaultPubSubRealtimeClient` subclasses + # `DefaultPubSubHttpClient`, so the same HTTP mock serves it, and this mirrors that + # suite's `RSL2a/returns-paginated-result-0` and `RSL2b/query-parameters-0`. channel_name = f'test-RTL10a-{random_id()}' captured_requests = [] @@ -86,7 +86,7 @@ async def test_rtl10a_supports_rest_params(): assert captured_requests[0].url.path == f'/channels/{channel_name}/messages' - # RTL10a: the parameters `RestChannel#history` takes + # RTL10a: the parameters `HttpChannel#history` takes await channel.history(direction='forwards', limit=50, start=1000, end=2000) query_params = captured_requests[1].url.query_params diff --git a/test/uts/realtime/unit/channels/channel_message_versions_test.py b/test/uts/realtime/unit/channels/channel_message_versions_test.py index ffac77fd..c8ea4a5f 100644 --- a/test/uts/realtime/unit/channels/channel_message_versions_test.py +++ b/test/uts/realtime/unit/channels/channel_message_versions_test.py @@ -4,14 +4,14 @@ The specification points at uts/rest/unit/channel/message_versions.md (RSL14) rather than listing steps of its own, so its single Test ID becomes one test driving an -`AblyRealtime` over the HTTP mock and asserting the core observable of the derived +`DefaultPubSubRealtimeClient` over the HTTP mock and asserting the core observable of the derived REST suite: the endpoint the call reaches and the paginated versions it returns. """ import uuid -from ably.http.paginatedresult import PaginatedResult -from ably.types.message import Message, MessageAction +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.message import Message, MessageAction from test.uts.helpers.client import realtime_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/realtime/unit/channels/channel_options_test.py b/test/uts/realtime/unit/channels/channel_options_test.py index 43284aa8..3da5dd78 100644 --- a/test/uts/realtime/unit/channels/channel_options_test.py +++ b/test/uts/realtime/unit/channels/channel_options_test.py @@ -15,13 +15,13 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelmode import ChannelMode -from ably.types.channeloptions import ChannelOptions -from ably.types.channelstate import ChannelState -from ably.util.crypto import get_default_params -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelmode import ChannelMode +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.crypto import get_default_params +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, @@ -106,7 +106,7 @@ async def test_tb2d_options_with_modes(): async def test_tb3_with_cipher_key(): # DEVIATION: TB3's `withCipherKey` constructor is absent from `ChannelOptions` # (`ably/types/channeloptions.py`), which takes a `CipherParams` and offers no factory - # that builds one from a key. `ably.util.crypto.get_default_params({'key': key})` is the + # that builds one from a key. `ably.pubsub.util.crypto.get_default_params({'key': key})` is the # nearest equivalent, and it is not on ChannelOptions. options = ChannelOptions.with_cipher_key(CIPHER_KEY) @@ -261,7 +261,7 @@ async def test_rts5a_creates_derived_channel(): # `DeriveOptions`. `Channels.__getattr__` (`ably/rest/channel.py:408`) answers any # unknown attribute with a channel of that name, so the call raises # `TypeError: 'RealtimeChannel' object is not callable` rather than AttributeError. - from ably import DeriveOptions + from ably.pubsub.server import DeriveOptions base_channel_name = 'test-RTS5a' client = realtime_client() @@ -278,7 +278,7 @@ async def test_rts5a_creates_derived_channel(): @deviation async def test_rts5a1_filter_base64_encoded(): # DEVIATION: derived channels are absent; see the RTS5a test. - from ably import DeriveOptions + from ably.pubsub.server import DeriveOptions base_channel_name = 'test-RTS5a1' client = realtime_client() @@ -296,7 +296,7 @@ async def test_rts5a1_filter_base64_encoded(): @deviation async def test_rts5a2_derived_with_params(): # DEVIATION: derived channels are absent; see the RTS5a test. - from ably import DeriveOptions + from ably.pubsub.server import DeriveOptions base_channel_name = 'test-RTS5a2' client = realtime_client() @@ -323,7 +323,7 @@ async def test_rts5a2_derived_with_params(): async def test_rts5_get_derived_with_options(): # DEVIATION: derived channels are absent; see the RTS5a test. `attachOnSubscribe` is # absent too; see the TB4 test. - from ably import DeriveOptions + from ably.pubsub.server import DeriveOptions base_channel_name = 'test-RTS5' client = realtime_client() @@ -341,7 +341,7 @@ async def test_rts5_get_derived_with_options(): @deviation async def test_do2a_filter_attribute(): # DEVIATION: `DeriveOptions` is absent from the library, so the import fails. - from ably import DeriveOptions + from ably.pubsub.server import DeriveOptions derive_options = DeriveOptions(filter="name == 'event' && data.count > 10") diff --git a/test/uts/realtime/unit/channels/channel_properties_test.py b/test/uts/realtime/unit/channels/channel_properties_test.py index dfa0bcff..212f6f79 100644 --- a/test/uts/realtime/unit/channels/channel_properties_test.py +++ b/test/uts/realtime/unit/channels/channel_properties_test.py @@ -10,10 +10,10 @@ import asyncio -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/channels/channel_publish_pending_test.py b/test/uts/realtime/unit/channels/channel_publish_pending_test.py index aa5e6e20..337edfd2 100644 --- a/test/uts/realtime/unit/channels/channel_publish_pending_test.py +++ b/test/uts/realtime/unit/channels/channel_publish_pending_test.py @@ -15,11 +15,11 @@ import pytest -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.operations import PublishResult -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.operations import PublishResult +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( STATE_TIMEOUT, await_channel_state, diff --git a/test/uts/realtime/unit/channels/channel_publish_test.py b/test/uts/realtime/unit/channels/channel_publish_test.py index 73d4ac41..bd531b82 100644 --- a/test/uts/realtime/unit/channels/channel_publish_test.py +++ b/test/uts/realtime/unit/channels/channel_publish_test.py @@ -15,7 +15,7 @@ as a task. `RealtimeChannel.publish()` takes its arguments positionally; the keyword form the -specifications write, which `RestChannel.publish()` does accept, raises `ValueError` here. +specifications write, which `HttpChannel.publish()` does accept, raises `ValueError` here. """ import asyncio @@ -25,12 +25,12 @@ import msgpack import pytest -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.message import Message -from ably.types.operations import PublishResult -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.message import Message +from ably.pubsub.types.operations import PublishResult +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/channels/channel_server_initiated_detach_test.py b/test/uts/realtime/unit/channels/channel_server_initiated_detach_test.py index f531f587..fc681546 100644 --- a/test/uts/realtime/unit/channels/channel_server_initiated_detach_test.py +++ b/test/uts/realtime/unit/channels/channel_server_initiated_detach_test.py @@ -7,10 +7,10 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import await_connection_state, poll_until, realtime_client from test.uts.helpers.clock import FakeClock, settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/channels/channel_state_events_test.py b/test/uts/realtime/unit/channels/channel_state_events_test.py index 8d800808..eb5ad4ac 100644 --- a/test/uts/realtime/unit/channels/channel_state_events_test.py +++ b/test/uts/realtime/unit/channels/channel_state_events_test.py @@ -7,11 +7,11 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState, ChannelStateChange -from ably.types.flags import Flag -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState, ChannelStateChange +from ably.pubsub.types.flags import Flag +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/channels/channel_subscribe_test.py b/test/uts/realtime/unit/channels/channel_subscribe_test.py index 354ff5d3..4361a72e 100644 --- a/test/uts/realtime/unit/channels/channel_subscribe_test.py +++ b/test/uts/realtime/unit/channels/channel_subscribe_test.py @@ -8,10 +8,10 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, @@ -46,7 +46,7 @@ def message_protocol_message(channel_name, messages, **fields): def message_filter(**criteria): """The specification's `MessageFilter` (MFI1), built from `criteria`.""" - from ably.types.messagefilter import MessageFilter + from ably.pubsub.types.messagefilter import MessageFilter return MessageFilter(**criteria) @@ -285,7 +285,7 @@ def accept_attach(msg): # UTS: realtime/unit/RTL7h/no-attach-on-subscribe-0 @deviation async def test_rtl7h_no_attach_on_subscribe(): - from ably.types.channeloptions import ChannelOptions + from ably.pubsub.types.channeloptions import ChannelOptions channel_name = 'test-RTL7h' attach_messages = [] diff --git a/test/uts/realtime/unit/channels/channel_update_delete_message_test.py b/test/uts/realtime/unit/channels/channel_update_delete_message_test.py index 6af97755..80867ef0 100644 --- a/test/uts/realtime/unit/channels/channel_update_delete_message_test.py +++ b/test/uts/realtime/unit/channels/channel_update_delete_message_test.py @@ -12,11 +12,11 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.message import Message, MessageAction -from ably.types.operations import MessageOperation, UpdateDeleteResult -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.message import Message, MessageAction +from ably.pubsub.types.operations import MessageOperation, UpdateDeleteResult +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.mock_websocket import MockWebSocket, attached_message, connected_message diff --git a/test/uts/realtime/unit/channels/channel_when_state_test.py b/test/uts/realtime/unit/channels/channel_when_state_test.py index 76bd651e..252901f2 100644 --- a/test/uts/realtime/unit/channels/channel_when_state_test.py +++ b/test/uts/realtime/unit/channels/channel_when_state_test.py @@ -5,9 +5,9 @@ import asyncio -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/channels/channels_collection_test.py b/test/uts/realtime/unit/channels/channels_collection_test.py index 3498ee12..8d85a2df 100644 --- a/test/uts/realtime/unit/channels/channels_collection_test.py +++ b/test/uts/realtime/unit/channels/channels_collection_test.py @@ -1,6 +1,6 @@ """Derived from uts/realtime/unit/channels/channels_collection.md in ably/specification. -Spec points: RTS1, RTS2, RTS3a, RTS4a +Spec points: RTS1, RTS2, RTS3a, RTS4c, RTS4d, RTS4e The specification's `channels.exists(name)` and `channels.names` are spelled `name in channels` and iteration over the collection in ably-python, and @@ -11,13 +11,15 @@ import asyncio -from ably.realtime.channel import Channels as RealtimeChannels -from ably.realtime.channel import RealtimeChannel -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from test.uts.helpers.client import await_connection_state, poll_until, realtime_client -from test.uts.helpers.deviations import deviation +import pytest + +from ably.pubsub.realtime.channel import Channels as RealtimeChannels +from ably.pubsub.realtime.channel import RealtimeChannel +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.util.exceptions import AblyException +from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.mock_websocket import ( CONNECTED_MESSAGE, MockWebSocket, @@ -119,37 +121,35 @@ async def test_rts3a_subscript_operator_channel(): assert channel1.name == channel_name -# UTS: realtime/unit/RTS4a/release-removes-channel-0 -async def test_rts4a_release_removes_channel(): - channel_name = 'test-RTS4a' +# UTS: realtime/unit/RTS4c/release-nonexistent-noop-0 +async def test_rts4c_release_nonexistent_noop(): + channel_name = 'test-RTS4c-nonexistent' client = realtime_client() - client.channels.get(channel_name) - assert (channel_name in client.channels) is True - - # `release` is synchronous here, so there is nothing to await client.channels.release(channel_name) assert (channel_name in client.channels) is False -# UTS: realtime/unit/RTS4a/release-nonexistent-noop-1 -async def test_rts4a_release_nonexistent_noop(): - channel_name = 'test-RTS4a-nonexistent' +# UTS: realtime/unit/RTS4d/release-removes-channel-0 +async def test_rts4d_release_removes_channel(): + channel_name = 'test-RTS4d' client = realtime_client() + channel = client.channels.get(channel_name) + assert channel.state == ChannelState.INITIALIZED + assert (channel_name in client.channels) is True + client.channels.release(channel_name) assert (channel_name in client.channels) is False -# UTS: realtime/unit/RTS4a/release-detaches-attached-2 -@deviation -async def test_rts4a_release_detaches_attached(): - # DEVIATION: RTS4a has release detach the channel before dropping it. - # `Channels.release` (`ably/realtime/channel.py:1012`) only deletes the entry from - # its dict, so no DETACH is sent and the channel is left attached in the Ably service. - channel_name = 'test-RTS4a-attached' +async def connected_client(answer_detach): + """A connected client whose mock answers each ATTACH, and each DETACH if `answer_detach`. + + Returns the client and the list of messages it sends. + """ messages_from_client = [] mock_ws = MockWebSocket( @@ -159,33 +159,51 @@ async def test_rts4a_release_detaches_attached(): def on_message_from_client(msg): messages_from_client.append(msg) if msg.get('action') == ProtocolMessageAction.ATTACH: - mock_ws.send_to_client(attached_message(channel_name)) - elif msg.get('action') == ProtocolMessageAction.DETACH: - mock_ws.send_to_client(detached_message(channel_name)) + mock_ws.send_to_client(attached_message(msg['channel'])) + elif msg.get('action') == ProtocolMessageAction.DETACH and answer_detach: + mock_ws.send_to_client(detached_message(msg['channel'])) mock_ws.on_message_from_client = on_message_from_client client = realtime_client(mock_ws) client.connect() await await_connection_state(client, ConnectionState.CONNECTED) + return client, messages_from_client + +# UTS: realtime/unit/RTS4d/release-after-detach-1 +async def test_rts4d_release_after_detach(): + channel_name = 'test-RTS4d-detached' + client, _ = await connected_client(answer_detach=True) channel = client.channels.get(channel_name) - await asyncio.wait_for(channel.attach(), OPERATION_TIMEOUT) - assert channel.state == ChannelState.ATTACHED - state_before_release = channel.state + await asyncio.wait_for(channel.attach(), OPERATION_TIMEOUT) + await asyncio.wait_for(channel.detach(), OPERATION_TIMEOUT) + assert channel.state == ChannelState.DETACHED client.channels.release(channel_name) - # A release which detaches sends DETACH on a task, and the channel reaches - # DETACHED only once the mock's answer has been handled, which is several - # yields away rather than one - await poll_until(lambda: channel.state == ChannelState.DETACHED, OPERATION_TIMEOUT, - 'the released channel to detach') - assert state_before_release == ChannelState.ATTACHED assert (channel_name in client.channels) is False + + +# UTS: realtime/unit/RTS4e/release-attached-fails-0 +async def test_rts4e_release_attached_fails(): + channel_name = 'test-RTS4e-attached' + client, messages_from_client = await connected_client(answer_detach=False) + channel = client.channels.get(channel_name) + + await asyncio.wait_for(channel.attach(), OPERATION_TIMEOUT) + assert channel.state == ChannelState.ATTACHED + + with pytest.raises(AblyException) as exc_info: + client.channels.release(channel_name) + + assert exc_info.value.code == 90011 + assert exc_info.value.status_code == 400 + assert channel.state == ChannelState.ATTACHED + assert (channel_name in client.channels) is True + assert client.channels.get(channel_name) is channel detach_messages = [m for m in messages_from_client if m.get('action') == ProtocolMessageAction.DETACH] - assert len(detach_messages) == 1 - assert channel.state == ChannelState.DETACHED + assert len(detach_messages) == 0 # UTS: realtime/unit/RTS3a/get-after-release-new-3 diff --git a/test/uts/realtime/unit/channels/message_field_population_test.py b/test/uts/realtime/unit/channels/message_field_population_test.py index 8efcd7df..b358442e 100644 --- a/test/uts/realtime/unit/channels/message_field_population_test.py +++ b/test/uts/realtime/unit/channels/message_field_population_test.py @@ -3,8 +3,8 @@ Spec points: TM2a, TM2c, TM2f """ -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.client import ( await_connection_state, poll_until, diff --git a/test/uts/realtime/unit/client/realtime_client_test.py b/test/uts/realtime/unit/client/realtime_client_test.py index 2e765a4c..b680edd5 100644 --- a/test/uts/realtime/unit/client/realtime_client_test.py +++ b/test/uts/realtime/unit/client/realtime_client_test.py @@ -8,13 +8,13 @@ import pytest -from ably.realtime.channel import Channels, RealtimeChannel -from ably.realtime.connection import Connection, ConnectionState -from ably.rest.auth import Auth -from ably.rest.push import Push, PushAdmin -from ably.transport.websockettransport import ProtocolMessageAction -from ably.util.exceptions import AblyException -from ably.util.helper import get_random_id +from ably.pubsub.http.auth import Auth +from ably.pubsub.http.push import Push, PushAdmin +from ably.pubsub.realtime.channel import Channels, RealtimeChannel +from ably.pubsub.realtime.connection import Connection, ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.util.exceptions import AblyException +from ably.pubsub.util.helper import get_random_id from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.deviations import deviation @@ -144,16 +144,13 @@ async def test_rtc17_client_id_attribute(): assert client.client_id == 'explicit-client-id' # DEVIATION: the spec asserts `client.clientId == client.auth.clientId`. - # `AblyRealtime.client_id` reads the client options, while `Auth.client_id` is + # `DefaultPubSubRealtimeClient.client_id` reads the client options, while `Auth.client_id` is # held at None for a realtime client until the server confirms one in a CONNECTED # message, so the two disagree before the connection is established. assert client.auth.client_id is None # UTS: realtime/unit/RTC1a/echo-messages-option-0 -# DEVIATION: ably-python has no `echo_messages` option and sends no `echo` query -# parameter. See deviations.md. -@deviation async def test_rtc1a_echo_messages_option(): # RTC1a_1: echoMessages defaults to true mock_ws = MockWebSocket() diff --git a/test/uts/realtime/unit/client/realtime_request_test.py b/test/uts/realtime/unit/client/realtime_request_test.py index 3a0fe289..858e9df4 100644 --- a/test/uts/realtime/unit/client/realtime_request_test.py +++ b/test/uts/realtime/unit/client/realtime_request_test.py @@ -17,7 +17,7 @@ # UTS: realtime/unit/RTC9/request-proxies-rest-0 async def test_rtc9_request_proxies_rest(): # The specification directs uts/rest/unit/request.md (RSC19) at a realtime client - # in place of a REST one. `AblyRealtime` subclasses `AblyRest`, so the same HTTP + # in place of a REST one. `DefaultPubSubRealtimeClient` subclasses `DefaultPubSubHttpClient`, so the same HTTP # mock serves it, and this mirrors that suite's `RSC19f/supports-http-methods-0` # and `RSC19d/response-items-decoded-5`. captured_requests = [] diff --git a/test/uts/realtime/unit/client/realtime_stats_test.py b/test/uts/realtime/unit/client/realtime_stats_test.py index ad9d8a44..a4fd40be 100644 --- a/test/uts/realtime/unit/client/realtime_stats_test.py +++ b/test/uts/realtime/unit/client/realtime_stats_test.py @@ -3,7 +3,7 @@ Spec points: RTC5, RTC5a, RTC5b """ -from ably.http.paginatedresult import PaginatedResult +from ably.pubsub.request.paginatedresult import PaginatedResult from test.uts.helpers.client import realtime_client from test.uts.helpers.mock_http import MockHttpClient @@ -30,7 +30,7 @@ # UTS: realtime/unit/RTC5/stats-proxies-rest-0 async def test_rtc5_stats_proxies_rest(): # The specification directs uts/rest/unit/stats.md (RSC6) at a realtime client - # in place of a REST one. `AblyRealtime` subclasses `AblyRest`, so the same HTTP + # in place of a REST one. `DefaultPubSubRealtimeClient` subclasses `DefaultPubSubHttpClient`, so the same HTTP # mock serves it, and this mirrors that suite's `RSC6a/returns-paginated-stats-0`. captured_requests = [] diff --git a/test/uts/realtime/unit/client/realtime_time_test.py b/test/uts/realtime/unit/client/realtime_time_test.py index 64c1fa8e..162336ed 100644 --- a/test/uts/realtime/unit/client/realtime_time_test.py +++ b/test/uts/realtime/unit/client/realtime_time_test.py @@ -12,7 +12,7 @@ # UTS: realtime/unit/RTC6/time-proxies-rest-0 async def test_rtc6_time_proxies_rest(): # The specification directs uts/rest/unit/time.md (RSC16) at a realtime client - # in place of a REST one. `AblyRealtime` subclasses `AblyRest`, so the same + # in place of a REST one. `DefaultPubSubRealtimeClient` subclasses `DefaultPubSubHttpClient`, so the same # HTTP mock serves it, and this mirrors that suite's `RSC16/returns-server-time-0`. captured_requests = [] diff --git a/test/uts/realtime/unit/client/realtime_timeouts_test.py b/test/uts/realtime/unit/client/realtime_timeouts_test.py index 6b0d3eba..3a9c21d9 100644 --- a/test/uts/realtime/unit/client/realtime_timeouts_test.py +++ b/test/uts/realtime/unit/client/realtime_timeouts_test.py @@ -7,11 +7,11 @@ import pytest -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.util.exceptions import AblyException -from ably.util.helper import get_random_id +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.util.exceptions import AblyException +from ably.pubsub.util.helper import get_random_id from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import FakeClock, settle from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/realtime/unit/connection/auto_connect_test.py b/test/uts/realtime/unit/connection/auto_connect_test.py index 29e8e12f..f77c0c39 100644 --- a/test/uts/realtime/unit/connection/auto_connect_test.py +++ b/test/uts/realtime/unit/connection/auto_connect_test.py @@ -3,7 +3,7 @@ Spec points: RTN3 """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.mock_websocket import MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/connection/backoff_jitter_test.py b/test/uts/realtime/unit/connection/backoff_jitter_test.py index 35b865b5..eee5a880 100644 --- a/test/uts/realtime/unit/connection/backoff_jitter_test.py +++ b/test/uts/realtime/unit/connection/backoff_jitter_test.py @@ -12,9 +12,9 @@ exact rather than sampled. """ -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import FakeClock, settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/connection/connection_failures_test.py b/test/uts/realtime/unit/connection/connection_failures_test.py index b2ea02c7..ffbcd2c8 100644 --- a/test/uts/realtime/unit/connection/connection_failures_test.py +++ b/test/uts/realtime/unit/connection/connection_failures_test.py @@ -5,8 +5,8 @@ import time -from ably.realtime.connection import ConnectionState -from ably.types.tokendetails import TokenDetails +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.tokendetails import TokenDetails from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import FakeClock, settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/connection/connection_id_key_test.py b/test/uts/realtime/unit/connection/connection_id_key_test.py index adfcd9aa..32f1ecbc 100644 --- a/test/uts/realtime/unit/connection/connection_id_key_test.py +++ b/test/uts/realtime/unit/connection/connection_id_key_test.py @@ -3,8 +3,8 @@ Spec points: RTN8, RTN8a, RTN8b, RTN8d, RTN9, RTN9a, RTN9b, RTN9d """ -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import FakeClock, settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/connection/connection_open_failures_test.py b/test/uts/realtime/unit/connection/connection_open_failures_test.py index af1e7393..0224e5b1 100644 --- a/test/uts/realtime/unit/connection/connection_open_failures_test.py +++ b/test/uts/realtime/unit/connection/connection_open_failures_test.py @@ -5,8 +5,8 @@ import time -from ably.realtime.connection import ConnectionState -from ably.types.tokendetails import TokenDetails +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.tokendetails import TokenDetails from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import FakeClock, settle from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/realtime/unit/connection/connection_ping_test.py b/test/uts/realtime/unit/connection/connection_ping_test.py index 1c3aaad7..dc3abd62 100644 --- a/test/uts/realtime/unit/connection/connection_ping_test.py +++ b/test/uts/realtime/unit/connection/connection_ping_test.py @@ -12,8 +12,8 @@ import asyncio -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.client import ( await_connection_state, poll_until, diff --git a/test/uts/realtime/unit/connection/connection_recovery_test.py b/test/uts/realtime/unit/connection/connection_recovery_test.py index abc92660..9684513c 100644 --- a/test/uts/realtime/unit/connection/connection_recovery_test.py +++ b/test/uts/realtime/unit/connection/connection_recovery_test.py @@ -13,9 +13,9 @@ import asyncio import json -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.client import await_channel_state, await_connection_state, realtime_client from test.uts.helpers.clock import FakeClock from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/connection/error_reason_test.py b/test/uts/realtime/unit/connection/error_reason_test.py index 2ad7bca8..a0a3622e 100644 --- a/test/uts/realtime/unit/connection/error_reason_test.py +++ b/test/uts/realtime/unit/connection/error_reason_test.py @@ -3,8 +3,8 @@ Spec points: RTN25 """ -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import FakeClock, settle from test.uts.helpers.mock_websocket import MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/connection/fallback_hosts_test.py b/test/uts/realtime/unit/connection/fallback_hosts_test.py index bcad6f43..69294ba8 100644 --- a/test/uts/realtime/unit/connection/fallback_hosts_test.py +++ b/test/uts/realtime/unit/connection/fallback_hosts_test.py @@ -24,9 +24,9 @@ import httpx -from ably.realtime import connectionmanager -from ably.realtime.connection import ConnectionState -from ably.transport.defaults import Defaults +from ably.pubsub.realtime import connectionmanager +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.defaults import Defaults from test.uts.helpers.client import await_connection_state, poll_until, realtime_client from test.uts.helpers.mock_http import MockHttpClient from test.uts.helpers.mock_websocket import MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/connection/forwards_compatibility_test.py b/test/uts/realtime/unit/connection/forwards_compatibility_test.py index 78f287b1..fcf0a8db 100644 --- a/test/uts/realtime/unit/connection/forwards_compatibility_test.py +++ b/test/uts/realtime/unit/connection/forwards_compatibility_test.py @@ -8,9 +8,9 @@ fields as written and no separate method is needed. """ -from ably.realtime.channel import ChannelState -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.realtime.channel import ChannelState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.mock_websocket import HEARTBEAT_MESSAGE, MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/connection/heartbeat_test.py b/test/uts/realtime/unit/connection/heartbeat_test.py index 681da166..42b0efd5 100644 --- a/test/uts/realtime/unit/connection/heartbeat_test.py +++ b/test/uts/realtime/unit/connection/heartbeat_test.py @@ -11,7 +11,7 @@ import pytest -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import ( await_connection_state, poll_until, diff --git a/test/uts/realtime/unit/connection/server_initiated_reauth_test.py b/test/uts/realtime/unit/connection/server_initiated_reauth_test.py index 3b0fa5c4..7c24e0f2 100644 --- a/test/uts/realtime/unit/connection/server_initiated_reauth_test.py +++ b/test/uts/realtime/unit/connection/server_initiated_reauth_test.py @@ -5,9 +5,9 @@ import time -from ably.realtime.connection import ConnectionEvent, ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.tokendetails import TokenDetails +from ably.pubsub.realtime.connection import ConnectionEvent, ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.tokendetails import TokenDetails from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.mock_websocket import MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/connection/update_events_test.py b/test/uts/realtime/unit/connection/update_events_test.py index 34f51b09..cfe5790f 100644 --- a/test/uts/realtime/unit/connection/update_events_test.py +++ b/test/uts/realtime/unit/connection/update_events_test.py @@ -3,8 +3,8 @@ Spec points: RTN24 """ -from ably.realtime.connection import ConnectionState -from ably.types.connectionstate import ConnectionEvent +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.connectionstate import ConnectionEvent from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/connection/when_state_test.py b/test/uts/realtime/unit/connection/when_state_test.py index 63f2f9d0..d41c4fc1 100644 --- a/test/uts/realtime/unit/connection/when_state_test.py +++ b/test/uts/realtime/unit/connection/when_state_test.py @@ -5,7 +5,7 @@ import asyncio -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState from test.uts.helpers.client import STATE_TIMEOUT, await_connection_state, realtime_client from test.uts.helpers.clock import settle from test.uts.helpers.mock_websocket import MockWebSocket, connected_message diff --git a/test/uts/realtime/unit/presence/local_presence_map_test.py b/test/uts/realtime/unit/presence/local_presence_map_test.py index 9e2e5ce2..1048143a 100644 --- a/test/uts/realtime/unit/presence/local_presence_map_test.py +++ b/test/uts/realtime/unit/presence/local_presence_map_test.py @@ -10,8 +10,8 @@ test/uts/deviations.md. """ -from ably.realtime.presencemap import PresenceMap -from ably.types.presence import PresenceAction +from ably.pubsub.realtime.presencemap import PresenceMap +from ably.pubsub.types.presence import PresenceAction from test.uts.helpers.deviations import deviation from test.uts.helpers.presence import presence_message, subscribed_presence diff --git a/test/uts/realtime/unit/presence/presence_map_test.py b/test/uts/realtime/unit/presence/presence_map_test.py index 7277096e..8a09145c 100644 --- a/test/uts/realtime/unit/presence/presence_map_test.py +++ b/test/uts/realtime/unit/presence/presence_map_test.py @@ -11,7 +11,7 @@ test/uts/deviations.md. """ -from ably.types.presence import PresenceAction +from ably.pubsub.types.presence import PresenceAction from test.uts.helpers.presence import presence_map, presence_message, subscribed_presence diff --git a/test/uts/realtime/unit/presence/presence_sync_test.py b/test/uts/realtime/unit/presence/presence_sync_test.py index 2e291c7d..0d228862 100644 --- a/test/uts/realtime/unit/presence/presence_sync_test.py +++ b/test/uts/realtime/unit/presence/presence_sync_test.py @@ -12,7 +12,7 @@ from datetime import datetime, timezone -from ably.types.presence import PresenceAction +from ably.pubsub.types.presence import PresenceAction from test.uts.helpers.deviations import deviation from test.uts.helpers.presence import presence_map, presence_message, subscribed_presence diff --git a/test/uts/realtime/unit/presence/realtime_presence_channel_state_test.py b/test/uts/realtime/unit/presence/realtime_presence_channel_state_test.py index 39341669..15d50b36 100644 --- a/test/uts/realtime/unit/presence/realtime_presence_channel_state_test.py +++ b/test/uts/realtime/unit/presence/realtime_presence_channel_state_test.py @@ -19,13 +19,13 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.realtime.presence import RealtimePresence -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag -from ably.types.presence import PresenceAction -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.realtime.presence import RealtimePresence +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag +from ably.pubsub.types.presence import PresenceAction +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, connected_client, diff --git a/test/uts/realtime/unit/presence/realtime_presence_enter_test.py b/test/uts/realtime/unit/presence/realtime_presence_enter_test.py index 01ad53e5..9168d22a 100644 --- a/test/uts/realtime/unit/presence/realtime_presence_enter_test.py +++ b/test/uts/realtime/unit/presence/realtime_presence_enter_test.py @@ -19,12 +19,12 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag -from ably.types.presence import PresenceAction -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag +from ably.pubsub.types.presence import PresenceAction +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/presence/realtime_presence_get_test.py b/test/uts/realtime/unit/presence/realtime_presence_get_test.py index 78fc6712..d0afbf7c 100644 --- a/test/uts/realtime/unit/presence/realtime_presence_get_test.py +++ b/test/uts/realtime/unit/presence/realtime_presence_get_test.py @@ -19,11 +19,11 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/realtime/unit/presence/realtime_presence_history_test.py b/test/uts/realtime/unit/presence/realtime_presence_history_test.py index c9943c74..59b60675 100644 --- a/test/uts/realtime/unit/presence/realtime_presence_history_test.py +++ b/test/uts/realtime/unit/presence/realtime_presence_history_test.py @@ -14,10 +14,10 @@ import uuid -from ably.http.paginatedresult import PaginatedResult -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.presence import PresenceAction, PresenceMessage +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.presence import PresenceAction, PresenceMessage from test.uts.helpers.client import await_connection_state, realtime_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/realtime/unit/presence/realtime_presence_reentry_test.py b/test/uts/realtime/unit/presence/realtime_presence_reentry_test.py index 702dd77d..0de3607e 100644 --- a/test/uts/realtime/unit/presence/realtime_presence_reentry_test.py +++ b/test/uts/realtime/unit/presence/realtime_presence_reentry_test.py @@ -15,10 +15,10 @@ import asyncio import uuid -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag -from ably.types.presence import PresenceAction +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag +from ably.pubsub.types.presence import PresenceAction from test.uts.helpers.client import await_channel_state, connected_client, poll_until from test.uts.helpers.clock import settle from test.uts.helpers.deviations import deviation diff --git a/test/uts/realtime/unit/presence/realtime_presence_subscribe_test.py b/test/uts/realtime/unit/presence/realtime_presence_subscribe_test.py index cec42e4f..66b562d5 100644 --- a/test/uts/realtime/unit/presence/realtime_presence_subscribe_test.py +++ b/test/uts/realtime/unit/presence/realtime_presence_subscribe_test.py @@ -10,12 +10,12 @@ import uuid -from ably.realtime.connection import ConnectionState -from ably.transport.websockettransport import ProtocolMessageAction -from ably.types.channeloptions import ChannelOptions -from ably.types.channelstate import ChannelState -from ably.types.flags import Flag -from ably.types.presence import PresenceAction +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.transport.websockettransport import ProtocolMessageAction +from ably.pubsub.types.channeloptions import ChannelOptions +from ably.pubsub.types.channelstate import ChannelState +from ably.pubsub.types.flags import Flag +from ably.pubsub.types.presence import PresenceAction from test.uts.helpers.client import ( await_channel_state, await_connection_state, diff --git a/test/uts/rest/integration/auth_test.py b/test/uts/rest/integration/auth_test.py index eac29ee2..9d7c253d 100644 --- a/test/uts/rest/integration/auth_test.py +++ b/test/uts/rest/integration/auth_test.py @@ -17,8 +17,8 @@ import pytest -from ably.transport.defaults import Defaults -from ably.util.exceptions import AblyException +from ably.pubsub.transport.defaults import Defaults +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import sandbox_rest_client from test.uts.helpers.deviations import deviation from test.uts.helpers.sandbox import extract_key_name, extract_key_secret, generate_jwt, random_id diff --git a/test/uts/rest/integration/batch_presence_test.py b/test/uts/rest/integration/batch_presence_test.py index 2ccb356b..ecf2d93c 100644 --- a/test/uts/rest/integration/batch_presence_test.py +++ b/test/uts/rest/integration/batch_presence_test.py @@ -2,38 +2,26 @@ Spec points: RSC24, BGR2, BGF2 -DEVIATION: ably-python has no batch API. `AblyRest` exposes no `batch_presence`, and -the package defines neither `BatchResult` nor `BatchPresenceSuccessResult` / -`BatchPresenceFailureResult`; the word "batch" appears nowhere under `ably/`. Every -test here therefore departs from the specification and is gated behind -RUN_DEVIATIONS, against the same spelling -[test/uts/rest/unit/batch_presence_test.py](../unit/batch_presence_test.py) gates on — -`client.batch_presence([...])` giving a result with `success_count`, `failure_count` -and `results` — so that dropping the marker is the only change either tier needs when -RSC24 lands. - -The setup halves are real, and the responses they assert against were confirmed by -hand through `client.request('GET', '/presence', params={'channels': ...})` against -the sandbox: the server does return the `successCount` / `failureCount` / `results` -envelope the specification describes, with `code` 40160 and `statusCode` 401 for a -channel the key has no capability for. Two details of that confirmation are recorded -beside the assertions they bear on — the `presence` key the server omits for an empty -channel, and the presence members a closed connection takes with it. - -See [deviations.md](../../deviations.md): the gating under *Failing Tests* -> -*Unimplemented features*, the omitted `presence` key and the closed connection under -*UTS Spec Errors*, and `enterClient` on an anonymous connection under *Failing Tests* -> -*Auth*. +The server returns the `successCount` / `failureCount` / `results` envelope the +specification describes, with `code` 40160 and `statusCode` 401 for a channel the key +has no capability for. Two details of the server's responses are recorded beside the +assertions they bear on — the `presence` key the server omits for an empty channel, and +the presence members a closed connection takes with it. + +See [deviations.md](../../deviations.md): the omitted `presence` key and the closed +connection under *UTS Spec Errors*, and `enterClient` on an anonymous connection under +*Failing Tests* -> *Auth*. """ -from ably.realtime.connection import ConnectionState +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.types.batch import BatchPresenceFailureResult, BatchPresenceSuccessResult +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_connection_state, sandbox_realtime_client, sandbox_rest_client, wall_clock_poll_until, ) -from test.uts.helpers.deviations import deviation from test.uts.helpers.sandbox import random_id @@ -45,20 +33,6 @@ def result_for(result, channel_name): raise AssertionError(f'No batch result for channel {channel_name!r}') -def is_success_result(entry): - """The specifications' `entry IS BatchPresenceSuccessResult`. - - Neither result class exists to name, so a success is told apart from a failure by - which attribute it carries, as the unit tier does. - """ - return getattr(entry, 'presence', None) is not None - - -def is_failure_result(entry): - """The specifications' `entry IS BatchPresenceFailureResult`.""" - return getattr(entry, 'error', None) is not None - - def member_for(entry, client_id): """The specifications' `presence.find(m => m.clientId == client_id)`.""" for member in entry.presence: @@ -96,7 +70,6 @@ async def enter_members(realtime, channel_name, members): return channel -@deviation # UTS: rest/integration/RSC24/batch-presence-multiple-channels-0 async def test_rsc24_batch_presence_multiple_channels(sandbox, use_binary_protocol): channel_a_name = 'batch-presence-a-' + random_id() @@ -119,7 +92,7 @@ async def test_rsc24_batch_presence_multiple_channels(sandbox, use_binary_protoc result_a = result_for(result, channel_a_name) result_b = result_for(result, channel_b_name) - assert is_success_result(result_a) + assert isinstance(result_a, BatchPresenceSuccessResult) assert len(result_a.presence) == 2 client_ids_a = [member.client_id for member in result_a.presence] assert 'user-1' in client_ids_a @@ -127,13 +100,12 @@ async def test_rsc24_batch_presence_multiple_channels(sandbox, use_binary_protoc assert member_for(result_a, 'user-1').data == 'data-a1' - assert is_success_result(result_b) + assert isinstance(result_b, BatchPresenceSuccessResult) assert len(result_b.presence) == 1 assert result_b.presence[0].client_id == 'user-3' assert result_b.presence[0].data == 'data-b1' -@deviation # UTS: rest/integration/RSC24/restricted-key-channel-failure-1 async def test_rsc24_restricted_key_channel_failure(sandbox, use_binary_protocol): # The specification hard-codes "channel6" as the channel `keys[2]` is allowed. The @@ -165,7 +137,7 @@ async def test_rsc24_restricted_key_channel_failure(sandbox, use_binary_protocol async def one_member_on_the_allowed_channel(): result = await restricted_rest.batch_presence([allowed_channel, denied_channel]) success = result_for(result, allowed_channel) - return result if is_success_result(success) and len(success.presence) == 1 else None + return result if isinstance(success, BatchPresenceSuccessResult) and len(success.presence) == 1 else None result = await wall_clock_poll_until( one_member_on_the_allowed_channel, @@ -178,16 +150,16 @@ async def one_member_on_the_allowed_channel(): success = result_for(result, allowed_channel) failure = result_for(result, denied_channel) - assert is_success_result(success) + assert isinstance(success, BatchPresenceSuccessResult) assert len(success.presence) == 1 assert success.presence[0].client_id == 'member-1' - assert is_failure_result(failure) + assert isinstance(failure, BatchPresenceFailureResult) + assert isinstance(failure.error, AblyException) assert failure.error.code == 40160 assert failure.error.status_code == 401 -@deviation # UTS: rest/integration/RSC24/empty-channel-presence-2 async def test_rsc24_empty_channel_presence(sandbox, use_binary_protocol): empty_channel = 'batch-empty-' + random_id() @@ -210,9 +182,9 @@ async def test_rsc24_empty_channel_presence(sandbox, use_binary_protocol): # The server counts the empty channel a success and leaves `presence` out of its # result altogether rather than sending `[]`, so an implementation of BGR2 has to # default the field for the specification's assertion to hold. - assert is_success_result(empty_result) + assert isinstance(empty_result, BatchPresenceSuccessResult) assert len(empty_result.presence) == 0 - assert is_success_result(populated_result) + assert isinstance(populated_result, BatchPresenceSuccessResult) assert len(populated_result.presence) == 1 assert populated_result.presence[0].client_id == 'someone' diff --git a/test/uts/rest/integration/mutable_messages_test.py b/test/uts/rest/integration/mutable_messages_test.py index b632b628..225302af 100644 --- a/test/uts/rest/integration/mutable_messages_test.py +++ b/test/uts/rest/integration/mutable_messages_test.py @@ -3,11 +3,11 @@ Spec points: RSL1n, RSL11, RSL14, RSL15, RSAN1, RSAN2, RSAN3 """ -from ably.http.paginatedresult import PaginatedResult -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.message import Message, MessageAction -from ably.types.operations import MessageOperation, PublishResult, UpdateDeleteResult -from ably.util.exceptions import AblyException +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.message import Message, MessageAction +from ably.pubsub.types.operations import MessageOperation, PublishResult, UpdateDeleteResult +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until from test.uts.helpers.sandbox import random_id diff --git a/test/uts/rest/integration/pagination_test.py b/test/uts/rest/integration/pagination_test.py index d7cacd6d..984d3044 100644 --- a/test/uts/rest/integration/pagination_test.py +++ b/test/uts/rest/integration/pagination_test.py @@ -3,8 +3,8 @@ Spec points: TG1, TG2, TG3, TG4, TG5 """ -from ably.http.paginatedresult import PaginatedResult -from ably.types.message import Message +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.message import Message from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until from test.uts.helpers.sandbox import random_id diff --git a/test/uts/rest/integration/presence_test.py b/test/uts/rest/integration/presence_test.py index 7f2a013e..6a3a5ab5 100644 --- a/test/uts/rest/integration/presence_test.py +++ b/test/uts/rest/integration/presence_test.py @@ -5,10 +5,10 @@ import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.realtime.connection import ConnectionState -from ably.types.presence import Presence, PresenceAction, PresenceMessage -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.presence import Presence, PresenceAction, PresenceMessage +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_connection_state, sandbox_realtime_client, @@ -65,7 +65,7 @@ async def test_rsp1_access_presence_from_channel(sandbox, use_binary_protocol): assert presence is not None # NOTE: the spec asserts `presence IS RestPresence`. The class behind - # `channel.presence` in ably-python is `ably.types.presence.Presence`. + # `channel.presence` in ably-python is `ably.pubsub.types.presence.Presence`. assert isinstance(presence, Presence) diff --git a/test/uts/rest/integration/proxy/rest_fallback_test.py b/test/uts/rest/integration/proxy/rest_fallback_test.py index 27eb1727..aed99207 100644 --- a/test/uts/rest/integration/proxy/rest_fallback_test.py +++ b/test/uts/rest/integration/proxy/rest_fallback_test.py @@ -19,8 +19,8 @@ import pytest -from ably import AblyRest -from ably.util.exceptions import AblyException +from ably.pubsub.server import create_http_client +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until from test.uts.helpers.deviations import deviation from test.uts.helpers.sandbox import SANDBOX_ENDPOINT, random_id @@ -44,7 +44,7 @@ def token_auth_callback(api_key): it: the token arrives over a connection the proxy never sees. """ async def auth_callback(params): - inner_rest = AblyRest(key=api_key, endpoint=SANDBOX_ENDPOINT) + inner_rest = create_http_client(key=api_key, endpoint=SANDBOX_ENDPOINT) try: return await inner_rest.auth.request_token() finally: diff --git a/test/uts/rest/integration/publish_test.py b/test/uts/rest/integration/publish_test.py index b81ca9dc..30b817bc 100644 --- a/test/uts/rest/integration/publish_test.py +++ b/test/uts/rest/integration/publish_test.py @@ -5,8 +5,8 @@ import pytest -from ably.types.message import Message -from ably.util.exceptions import AblyException +from ably.pubsub.types.message import Message +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until from test.uts.helpers.sandbox import random_id diff --git a/test/uts/rest/integration/push_admin_test.py b/test/uts/rest/integration/push_admin_test.py index 5b4a5f45..16720dd3 100644 --- a/test/uts/rest/integration/push_admin_test.py +++ b/test/uts/rest/integration/push_admin_test.py @@ -6,8 +6,8 @@ import pytest -from ably import AblyException, DeviceDetails, PushChannelSubscription -from ably.http.paginatedresult import PaginatedResult +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.server import AblyException, DeviceDetails, PushChannelSubscription from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until from test.uts.helpers.sandbox import random_id diff --git a/test/uts/rest/integration/push_channels_test.py b/test/uts/rest/integration/push_channels_test.py index 36f0d551..79b60a65 100644 --- a/test/uts/rest/integration/push_channels_test.py +++ b/test/uts/rest/integration/push_channels_test.py @@ -4,11 +4,11 @@ DEVIATION: ably-python implements neither the PushChannel interface (RSH7, the `push` field on a channel) nor LocalDevice (RSH8). `ably/rest/channel.py` gives a channel no -`push`, `AblyRest` no `device`, and `ably/types/device.py` defines only `DeviceDetails`. +`push`, `DefaultPubSubHttpClient` no `device`, and `ably/types/device.py` defines only `DeviceDetails`. Both tests here therefore depart from the specification and are gated behind RUN_DEVIATIONS, against the same spelling [test/uts/rest/unit/push/push_channels_test.py](../unit/push/push_channels_test.py) -gates on — `ably.types.device.LocalDevice`, `client.device` and +gates on — `ably.pubsub.types.device.LocalDevice`, `client.device` and `channel.push.subscribe_device()` and friends — so that dropping the marker is the only change either tier needs when RSH7 lands. @@ -31,7 +31,7 @@ Errors*. """ -from ably import AblyException, DeviceDetails +from ably.pubsub.server import AblyException, DeviceDetails from test.uts.helpers.client import sandbox_rest_client, wall_clock_poll_until from test.uts.helpers.deviations import deviation from test.uts.helpers.sandbox import random_id @@ -55,7 +55,7 @@ def set_local_device(client, device_id, device_identity_token=PLACEHOLDER_DEVICE The import is deliberately inside the call, so that a file-level import of a name that does not exist does not take the collection of the whole package down with it. """ - from ably.types.device import LocalDevice + from ably.pubsub.types.device import LocalDevice client.device = LocalDevice( id=device_id, diff --git a/test/uts/rest/integration/revoke_tokens_test.py b/test/uts/rest/integration/revoke_tokens_test.py index 7b2adacd..c83a5da1 100644 --- a/test/uts/rest/integration/revoke_tokens_test.py +++ b/test/uts/rest/integration/revoke_tokens_test.py @@ -27,8 +27,8 @@ import pytest -from ably.realtime.connection import ConnectionState -from ably.util.exceptions import AblyException +from ably.pubsub.realtime.connection import ConnectionState +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import ( await_connection_state, next_connection_state, diff --git a/test/uts/rest/integration/time_stats_test.py b/test/uts/rest/integration/time_stats_test.py index 39d6e465..b2d3abb7 100644 --- a/test/uts/rest/integration/time_stats_test.py +++ b/test/uts/rest/integration/time_stats_test.py @@ -8,9 +8,9 @@ import pytest_asyncio -from ably import AblyRest -from ably.http.paginatedresult import PaginatedResult -from ably.types.stats import Stats +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.server import create_http_client +from ably.pubsub.types.stats import Stats from test.uts.helpers.client import sandbox_rest_client from test.uts.helpers.sandbox import SANDBOX_ENDPOINT @@ -48,8 +48,8 @@ async def app_with_stats(sandbox): """ interval = (datetime.now(timezone.utc).replace(tzinfo=None) - STATS_INTERVAL_AGE).replace( second=0, microsecond=0) - client = AblyRest(key=sandbox.key(0).key_str, endpoint=SANDBOX_ENDPOINT, - use_binary_protocol=False) + client = create_http_client(key=sandbox.key(0).key_str, endpoint=SANDBOX_ENDPOINT, + use_binary_protocol=False) try: await client.http.post('/stats', body=[{ 'intervalId': Stats.to_interval_id(interval, 'minute'), diff --git a/test/uts/rest/unit/auth/auth_callback_test.py b/test/uts/rest/unit/auth/auth_callback_test.py index 235abbb0..4511aefe 100644 --- a/test/uts/rest/unit/auth/auth_callback_test.py +++ b/test/uts/rest/unit/auth/auth_callback_test.py @@ -9,10 +9,10 @@ import pytest -from ably import api_version -from ably.types.tokendetails import TokenDetails -from ably.types.tokenrequest import TokenRequest -from ably.util.exceptions import AblyException +from ably.pubsub.server import api_version +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.types.tokenrequest import TokenRequest +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/auth/auth_scheme_test.py b/test/uts/rest/unit/auth/auth_scheme_test.py index 30ed44cc..e69b732a 100644 --- a/test/uts/rest/unit/auth/auth_scheme_test.py +++ b/test/uts/rest/unit/auth/auth_scheme_test.py @@ -8,9 +8,9 @@ import pytest -from ably import api_version -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException +from ably.pubsub.server import api_version +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import spec_error from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/auth/authorize_test.py b/test/uts/rest/unit/auth/authorize_test.py index a280fc2a..5d1f50d3 100644 --- a/test/uts/rest/unit/auth/authorize_test.py +++ b/test/uts/rest/unit/auth/authorize_test.py @@ -9,9 +9,9 @@ import pytest -from ably.types.authoptions import AuthOptions -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException +from ably.pubsub.types.authoptions import AuthOptions +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation, spec_error from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/auth/client_id_test.py b/test/uts/rest/unit/auth/client_id_test.py index 5dcc536b..58a2990e 100644 --- a/test/uts/rest/unit/auth/client_id_test.py +++ b/test/uts/rest/unit/auth/client_id_test.py @@ -8,8 +8,8 @@ import pytest -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException, IncompatibleClientIdException +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException, IncompatibleClientIdException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/auth/revoke_tokens_test.py b/test/uts/rest/unit/auth/revoke_tokens_test.py index f8a02085..0adb05f9 100644 --- a/test/uts/rest/unit/auth/revoke_tokens_test.py +++ b/test/uts/rest/unit/auth/revoke_tokens_test.py @@ -17,7 +17,7 @@ import msgpack import pytest -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/auth/token_details_test.py b/test/uts/rest/unit/auth/token_details_test.py index e7e1bc88..d4031743 100644 --- a/test/uts/rest/unit/auth/token_details_test.py +++ b/test/uts/rest/unit/auth/token_details_test.py @@ -7,10 +7,10 @@ import pytest -from ably.types.authoptions import AuthOptions -from ably.types.capability import Capability -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException +from ably.pubsub.types.authoptions import AuthOptions +from ably.pubsub.types.capability import Capability +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/auth/token_renewal_test.py b/test/uts/rest/unit/auth/token_renewal_test.py index 6f4f6243..2918d1a1 100644 --- a/test/uts/rest/unit/auth/token_renewal_test.py +++ b/test/uts/rest/unit/auth/token_renewal_test.py @@ -9,8 +9,8 @@ import msgpack import pytest -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import spec_error from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/batch_presence_test.py b/test/uts/rest/unit/batch_presence_test.py index 187d61ad..e1c67fc0 100644 --- a/test/uts/rest/unit/batch_presence_test.py +++ b/test/uts/rest/unit/batch_presence_test.py @@ -2,28 +2,16 @@ Spec points: RSC24, BAR2, BGR2, BGF2 -NOTE: ably-python has no batch API. `AblyRest` exposes no `batch_presence`, and the package -defines neither `BatchResult`/`BatchPresenceResponse` nor `BatchPresenceSuccessResult`/ -`BatchPresenceFailureResult`; the word "batch" appears nowhere under `ably/`. Every test in -this file therefore departs from the specification and is gated behind `RUN_DEVIATIONS`. -Each carries the assertion the spec calls for, written against the name ably-python would -use once RSC24 is implemented, so that dropping the `@deviation` marker is the only change -needed when it is. Today a batch presence query has to be hand-rolled by the caller through -`client.request('GET', '/presence', version=..., params={'channels': ...})`, which returns -an `HttpPaginatedResponse` of raw dicts rather than decoded `PresenceMessage`s, and which -does not raise on an error status. - -Counts are read as `success_count` / `failure_count`, the snake_case spelling of BAR2a and -BAR2b, and a success result is told apart from a failure result by which attributes it -carries rather than by `isinstance`, since neither class exists to name. +The specification's `BatchPresenceResponse` is ably-python's `BatchResult`, whose counts are +read as `success_count` / `failure_count`, the snake_case spelling of BAR2a and BAR2b. """ import pytest -from ably.types.presence import PresenceAction -from ably.util.exceptions import AblyException +from ably.pubsub.types.batch import BatchPresenceFailureResult, BatchPresenceSuccessResult +from ably.pubsub.types.presence import PresenceAction +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client -from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient @@ -40,7 +28,6 @@ def respond_with(status, body): # UTS: rest/unit/RSC24/get-presence-channels-param-0 -@deviation async def test_rsc24_batch_presence_get_presence_channels_param(): captured_requests = [] mock_http = MockHttpClient( @@ -65,7 +52,6 @@ async def test_rsc24_batch_presence_get_presence_channels_param(): # UTS: rest/unit/RSC24/single-channel-param-0 -@deviation async def test_rsc24_batch_presence_single_channel_param(): captured_requests = [] mock_http = MockHttpClient( @@ -86,7 +72,6 @@ async def test_rsc24_batch_presence_single_channel_param(): # UTS: rest/unit/RSC24/special-chars-comma-joined-0 -@deviation async def test_rsc24_batch_presence_special_chars_comma_joined(): captured_requests = [] mock_http = MockHttpClient( @@ -108,7 +93,6 @@ async def test_rsc24_batch_presence_special_chars_comma_joined(): # UTS: rest/unit/BAR2/mixed-success-failure-counts-0 -@deviation async def test_bar2_batch_presence_mixed_success_failure_counts(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -136,7 +120,6 @@ async def test_bar2_batch_presence_mixed_success_failure_counts(): # UTS: rest/unit/BAR2/all-success-counts-0 -@deviation async def test_bar2_batch_presence_all_success_counts(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -159,7 +142,6 @@ async def test_bar2_batch_presence_all_success_counts(): # UTS: rest/unit/BAR2/all-failure-counts-0 -@deviation async def test_bar2_batch_presence_all_failure_counts(): error = {'code': 40160, 'statusCode': 401, 'message': 'Not permitted'} mock_http = MockHttpClient( @@ -183,7 +165,6 @@ async def test_bar2_batch_presence_all_failure_counts(): # UTS: rest/unit/BGR2/success-with-members-0 -@deviation async def test_bgr2_batch_presence_success_with_members(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -222,7 +203,7 @@ async def test_bgr2_batch_presence_success_with_members(): assert len(result.results) == 1 success = result.results[0] - assert getattr(success, 'presence', None) is not None + assert isinstance(success, BatchPresenceSuccessResult) assert success.channel == 'my-channel' assert len(success.presence) == 2 @@ -237,7 +218,6 @@ async def test_bgr2_batch_presence_success_with_members(): # UTS: rest/unit/BGR2/success-empty-presence-0 -@deviation async def test_bgr2_batch_presence_success_empty_presence(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -254,13 +234,12 @@ async def test_bgr2_batch_presence_success_empty_presence(): result = await client.batch_presence(['empty-channel']) success = result.results[0] - assert getattr(success, 'presence', None) is not None + assert isinstance(success, BatchPresenceSuccessResult) assert success.channel == 'empty-channel' assert len(success.presence) == 0 # UTS: rest/unit/BGF2/failure-error-details-0 -@deviation async def test_bgf2_batch_presence_failure_error_details(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -286,7 +265,8 @@ async def test_bgf2_batch_presence_failure_error_details(): assert len(result.results) == 1 failure = result.results[0] - assert getattr(failure, 'error', None) is not None + assert isinstance(failure, BatchPresenceFailureResult) + assert isinstance(failure.error, AblyException) assert failure.channel == 'restricted-channel' assert failure.error.code == 40160 assert failure.error.status_code == 401 @@ -294,7 +274,6 @@ async def test_bgf2_batch_presence_failure_error_details(): # UTS: rest/unit/RSC24/mixed-success-failure-results-0 -@deviation async def test_rsc24_batch_presence_mixed_success_failure_results(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -329,18 +308,17 @@ async def test_rsc24_batch_presence_mixed_success_failure_results(): assert result.failure_count == 1 assert len(result.results) == 2 - assert getattr(result.results[0], 'presence', None) is not None + assert isinstance(result.results[0], BatchPresenceSuccessResult) assert result.results[0].channel == 'allowed-channel' assert len(result.results[0].presence) == 1 assert result.results[0].presence[0].client_id == 'user-1' - assert getattr(result.results[1], 'error', None) is not None + assert isinstance(result.results[1], BatchPresenceFailureResult) assert result.results[1].channel == 'restricted-channel' assert result.results[1].error.code == 40160 # UTS: rest/unit/RSC24/server-error-propagated-0 -@deviation async def test_rsc24_batch_presence_server_error_propagated(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -358,7 +336,6 @@ async def test_rsc24_batch_presence_server_error_propagated(): # UTS: rest/unit/RSC24/auth-error-propagated-0 -@deviation async def test_rsc24_batch_presence_auth_error_propagated(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -376,7 +353,6 @@ async def test_rsc24_batch_presence_auth_error_propagated(): # UTS: rest/unit/RSC24/uses-configured-auth-0 -@deviation async def test_rsc24_batch_presence_uses_configured_auth(): captured_requests = [] mock_http = MockHttpClient( diff --git a/test/uts/rest/unit/batch_publish_test.py b/test/uts/rest/unit/batch_publish_test.py index 338de3ac..bf40f99d 100644 --- a/test/uts/rest/unit/batch_publish_test.py +++ b/test/uts/rest/unit/batch_publish_test.py @@ -2,31 +2,30 @@ Spec points: RSC22c, RSC22d, BSP2a, BSP2b, BPR2a, BPR2b, BPR2c, BPF2a, BPF2b -NOTE: ably-python has no batch API. `AblyRest` exposes no `batch_publish`, and the package -defines neither `BatchPublishSpec` nor `BatchResult`/`BatchPublishSuccessResult`/ -`BatchPublishFailureResult`; the word "batch" appears nowhere under `ably/`. Every test in -this file therefore departs from the specification and is gated behind `RUN_DEVIATIONS`. -Each carries the assertion the spec calls for, written against the name ably-python would -use once RSC22 is implemented, so that dropping the `@deviation` marker is the only change -needed when it is. Today a batch publish has to be hand-rolled by the caller through -`client.request('POST', '/messages', version=..., body=...)`, which does no spec -construction, no RSL4 message encoding and no RSL1k1 idempotent ID generation. - -Because `BatchPublishSpec` does not exist to construct, a spec is written as a mapping of -the BSP2 attributes. Results are read by attribute, as the spec writes them, and a success -result is told apart from a failure result by which attributes it carries rather than by -`isinstance`, since neither class exists to name. +The specification's mocks answer in the legacy response format, a flat array of per-channel +results, which the server sends to a client sending `X-Ably-Version: 2` or no version at all. +ably-python sends version 5, and to that the server answers every batch publish with HTTP 201 +and an array holding a `BatchResult` envelope for each spec sent, a single spec sent as a bare +object included. Every fixture is corrected to that shape through `batch_result()`, and the +assertions stand as the specification writes them. See *Batch response envelopes disagree +between sibling specs* in [deviations.md](../../deviations.md). """ +import json import uuid import msgpack import pytest -from ably.types.message import Message -from ably.util.exceptions import AblyException +from ably.pubsub.types.batch import ( + BatchPublishFailureResult, + BatchPublishSpec, + BatchPublishSuccessResult, + BatchResult, +) +from ably.pubsub.types.message import Message +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client -from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient @@ -37,7 +36,7 @@ def random_id(): def capture_and_respond(captured_requests, status=201, body=None): def on_request(request): captured_requests.append(request) - request.respond_with(status, body if body is not None else []) + request.respond_with(status, body if body is not None else success_response(request)) return on_request @@ -50,8 +49,29 @@ def success_result(channel, message_id='msg', serials=('s1',)): return {'channel': channel, 'messageId': message_id, 'serials': list(serials)} +def batch_result(*results): + """One spec's `BatchResult` envelope, holding the per-channel `results` given. + + UTS SPEC ERROR: batch_publish.md - the specification's mocks send the per-channel results + as a flat array, the legacy format, where the server answers ably-python's + `X-Ably-Version: 5` with an array of these envelopes, one per spec. + """ + failure_count = sum(1 for result in results if 'error' in result) + return { + 'successCount': len(results) - failure_count, + 'failureCount': failure_count, + 'results': list(results), + } + + +def success_response(request): + """What the server answers a batch publish with when every channel succeeds.""" + body = msgpack.unpackb(request.body) + specs = body if isinstance(body, list) else [body] + return [batch_result(*(success_result(channel) for channel in spec['channels'])) for spec in specs] + + # UTS: rest/unit/RSC22c/single-spec-post-messages-0 -@deviation async def test_rsc22c_batch_publish_single_spec_post_messages(): channel_name_1 = f'test-RSC22c1-a-{random_id()}' channel_name_2 = f'test-RSC22c1-b-{random_id()}' @@ -63,10 +83,10 @@ async def test_rsc22c_batch_publish_single_spec_post_messages(): ) client = rest_client(mock_http) - await client.batch_publish({ - 'channels': [channel_name_1, channel_name_2], - 'messages': [Message(name='event', data='hello')], - }) + await client.batch_publish(BatchPublishSpec( + channels=[channel_name_1, channel_name_2], + messages=[Message(name='event', data='hello')], + )) assert len(captured_requests) == 1 request = captured_requests[0] @@ -81,7 +101,6 @@ async def test_rsc22c_batch_publish_single_spec_post_messages(): # UTS: rest/unit/RSC22c/array-specs-post-messages-0 -@deviation async def test_rsc22c_batch_publish_array_specs_post_messages(): channel_name_1 = f'test-RSC22c2-a-{random_id()}' channel_name_2 = f'test-RSC22c2-b-{random_id()}' @@ -94,8 +113,8 @@ async def test_rsc22c_batch_publish_array_specs_post_messages(): client = rest_client(mock_http) await client.batch_publish([ - {'channels': [channel_name_1], 'messages': [Message(name='e1', data='d1')]}, - {'channels': [channel_name_2], 'messages': [Message(name='e2', data='d2')]}, + BatchPublishSpec(channels=[channel_name_1], messages=[Message(name='e1', data='d1')]), + BatchPublishSpec(channels=[channel_name_2], messages=[Message(name='e2', data='d2')]), ]) assert len(captured_requests) == 1 @@ -113,13 +132,13 @@ async def test_rsc22c_batch_publish_array_specs_post_messages(): # UTS: rest/unit/RSC22c/single-spec-single-result-0 -@deviation async def test_rsc22c_batch_publish_single_spec_single_result(): channel_name = f'test-RSC22c3-{random_id()}' # UTS SPEC ERROR: RSC22c3 - the mock body is a bare result object, but RSC22b says the REST # response "will still be an array", from which the single-spec overload extracts one element. - response_body = [{'channel': channel_name, 'messageId': 'msg123', 'serials': ['serial1']}] + # The server sends that element as the spec's BatchResult envelope. + response_body = [batch_result({'channel': channel_name, 'messageId': 'msg123', 'serials': ['serial1']})] mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -127,26 +146,25 @@ async def test_rsc22c_batch_publish_single_spec_single_result(): ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) - assert not isinstance(result, list) + assert isinstance(result, BatchResult) assert len(result.results) == 1 assert result.results[0].channel == channel_name assert result.results[0].message_id == 'msg123' # UTS: rest/unit/RSC22c/array-specs-array-results-0 -@deviation async def test_rsc22c_batch_publish_array_specs_array_results(): channel_name_1 = f'test-RSC22c4-a-{random_id()}' channel_name_2 = f'test-RSC22c4-b-{random_id()}' response_body = [ - success_result(channel_name_1, 'msg1', ['s1']), - success_result(channel_name_2, 'msg2', ['s2']), + batch_result(success_result(channel_name_1, 'msg1', ['s1'])), + batch_result(success_result(channel_name_2, 'msg2', ['s2'])), ] mock_http = MockHttpClient( @@ -156,28 +174,28 @@ async def test_rsc22c_batch_publish_array_specs_array_results(): client = rest_client(mock_http) results = await client.batch_publish([ - {'channels': [channel_name_1], 'messages': [Message(name='e1', data='d1')]}, - {'channels': [channel_name_2], 'messages': [Message(name='e2', data='d2')]}, + BatchPublishSpec(channels=[channel_name_1], messages=[Message(name='e1', data='d1')]), + BatchPublishSpec(channels=[channel_name_2], messages=[Message(name='e2', data='d2')]), ]) assert isinstance(results, list) assert len(results) == 2 + assert all(isinstance(result, BatchResult) for result in results) assert results[0].results[0].channel == channel_name_1 assert results[1].results[0].channel == channel_name_2 # UTS: rest/unit/RSC22c/multiple-channels-multiple-results-0 -@deviation async def test_rsc22c_batch_publish_multiple_channels_multiple_results(): channel_name_1 = f'test-RSC22c5-a-{random_id()}' channel_name_2 = f'test-RSC22c5-b-{random_id()}' channel_name_3 = f'test-RSC22c5-c-{random_id()}' - response_body = [ + response_body = [batch_result( success_result(channel_name_1, 'msg1', ['s1']), success_result(channel_name_2, 'msg2', ['s2']), success_result(channel_name_3, 'msg3', ['s3']), - ] + )] mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -185,17 +203,16 @@ async def test_rsc22c_batch_publish_multiple_channels_multiple_results(): ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name_1, channel_name_2, channel_name_3], - 'messages': [Message(name='event', data='hello')], - }) + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name_1, channel_name_2, channel_name_3], + messages=[Message(name='event', data='hello')], + )) assert len(result.results) == 3 assert [entry.channel for entry in result.results] == [channel_name_1, channel_name_2, channel_name_3] # UTS: rest/unit/RSC22c/messages-encoded-per-rsl4-0 -@deviation async def test_rsc22c_batch_publish_messages_encoded_per_rsl4(): channel_name = f'test-RSC22c6-{random_id()}' @@ -206,14 +223,14 @@ async def test_rsc22c_batch_publish_messages_encoded_per_rsl4(): ) client = rest_client(mock_http) - await client.batch_publish({ - 'channels': [channel_name], - 'messages': [ + await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[ Message(name='string', data='plain text'), Message(name='binary', data=b'\x01\x02\x03'), Message(name='json', data={'key': 'value'}), ], - }) + )) assert len(captured_requests) == 1 messages = msgpack.unpackb(captured_requests[0].body)['messages'] @@ -228,12 +245,11 @@ async def test_rsc22c_batch_publish_messages_encoded_per_rsl4(): assert 'encoding' not in messages[1] or messages[1]['encoding'] is None # RSL4c3: a JSON payload is stringified and the encoding attribute is set to "json" - assert messages[2]['data'] == '{"key":"value"}' + assert json.loads(messages[2]['data']) == {'key': 'value'} assert messages[2]['encoding'] == 'json' # UTS: rest/unit/RSC22c/uses-configured-auth-0 -@deviation async def test_rsc22c_batch_publish_uses_configured_auth(): channel_name = f'test-RSC22c7-{random_id()}' @@ -244,10 +260,10 @@ async def test_rsc22c_batch_publish_uses_configured_auth(): ) token_client = rest_client(mock_http, token='fake-token') - await token_client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + await token_client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) assert len(captured_requests) == 1 assert captured_requests[0].headers['Authorization'].startswith('Bearer ') @@ -258,17 +274,16 @@ async def test_rsc22c_batch_publish_uses_configured_auth(): mock_http.on_request = capture_and_respond(basic_requests) basic_client = rest_client(mock_http) - await basic_client.batch_publish({ - 'channels': [basic_channel_name], - 'messages': [Message(name='event', data='hello')], - }) + await basic_client.batch_publish(BatchPublishSpec( + channels=[basic_channel_name], + messages=[Message(name='event', data='hello')], + )) assert len(basic_requests) == 1 assert basic_requests[0].headers['Authorization'].startswith('Basic ') # UTS: rest/unit/RSC22d/idempotent-ids-generated-0 -@deviation async def test_rsc22d_batch_publish_idempotent_ids_generated(): captured_requests = [] mock_http = MockHttpClient( @@ -278,14 +293,14 @@ async def test_rsc22d_batch_publish_idempotent_ids_generated(): client = rest_client(mock_http, idempotent_rest_publishing=True) await client.batch_publish([ - { - 'channels': [f'test-RSC22d-a-{random_id()}'], - 'messages': [Message(name='e1', data='d1'), Message(name='e2', data='d2')], - }, - { - 'channels': [f'test-RSC22d-b-{random_id()}'], - 'messages': [Message(name='e3', data='d3'), Message(name='e4', data='d4')], - }, + BatchPublishSpec( + channels=[f'test-RSC22d-a-{random_id()}'], + messages=[Message(name='e1', data='d1'), Message(name='e2', data='d2')], + ), + BatchPublishSpec( + channels=[f'test-RSC22d-b-{random_id()}'], + messages=[Message(name='e3', data='d3'), Message(name='e4', data='d4')], + ), ]) assert len(captured_requests) == 1 @@ -307,7 +322,6 @@ async def test_rsc22d_batch_publish_idempotent_ids_generated(): # UTS: rest/unit/RSC22d/explicit-ids-preserved-0 -@deviation async def test_rsc22d_batch_publish_explicit_ids_preserved(): channel_name = f'test-RSC22d-explicit-{random_id()}' @@ -318,13 +332,13 @@ async def test_rsc22d_batch_publish_explicit_ids_preserved(): ) client = rest_client(mock_http, idempotent_rest_publishing=True) - await client.batch_publish({ - 'channels': [channel_name], - 'messages': [ + await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[ Message(name='e1', data='d1', id='explicit-id-1'), Message(name='e2', data='d2', id='explicit-id-2'), ], - }) + )) assert len(captured_requests) == 1 messages = msgpack.unpackb(captured_requests[0].body)['messages'] @@ -334,7 +348,6 @@ async def test_rsc22d_batch_publish_explicit_ids_preserved(): # UTS: rest/unit/RSC22d/ids-not-generated-disabled-0 -@deviation async def test_rsc22d_batch_publish_ids_not_generated_disabled(): channel_name = f'test-RSC22d-disabled-{random_id()}' @@ -345,10 +358,10 @@ async def test_rsc22d_batch_publish_ids_not_generated_disabled(): ) client = rest_client(mock_http, idempotent_rest_publishing=False) - await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='e1', data='d1'), Message(name='e2', data='d2')], - }) + await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='e1', data='d1'), Message(name='e2', data='d2')], + )) assert len(captured_requests) == 1 messages = msgpack.unpackb(captured_requests[0].body)['messages'] @@ -358,7 +371,6 @@ async def test_rsc22d_batch_publish_ids_not_generated_disabled(): # UTS: rest/unit/BSP2a/channels-array-strings-0 -@deviation async def test_bsp2a_batch_publish_spec_channels_array_strings(): channel_name_1 = f'test-BSP2a-a-{random_id()}' channel_name_2 = f'test-BSP2a-b-{random_id()}' @@ -371,10 +383,10 @@ async def test_bsp2a_batch_publish_spec_channels_array_strings(): ) client = rest_client(mock_http) - await client.batch_publish({ - 'channels': [channel_name_1, channel_name_2, channel_name_3], - 'messages': [Message(name='event', data='hello')], - }) + await client.batch_publish(BatchPublishSpec( + channels=[channel_name_1, channel_name_2, channel_name_3], + messages=[Message(name='event', data='hello')], + )) assert len(captured_requests) == 1 channels = msgpack.unpackb(captured_requests[0].body)['channels'] @@ -385,7 +397,6 @@ async def test_bsp2a_batch_publish_spec_channels_array_strings(): # UTS: rest/unit/BSP2b/messages-array-objects-0 -@deviation async def test_bsp2b_batch_publish_spec_messages_array_objects(): channel_name = f'test-BSP2b-{random_id()}' @@ -396,13 +407,13 @@ async def test_bsp2b_batch_publish_spec_messages_array_objects(): ) client = rest_client(mock_http) - await client.batch_publish({ - 'channels': [channel_name], - 'messages': [ + await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[ Message(name='event1', data='data1'), Message(name='event2', data={'key': 'value'}), ], - }) + )) assert len(captured_requests) == 1 messages = msgpack.unpackb(captured_requests[0].body)['messages'] @@ -415,118 +426,117 @@ async def test_bsp2b_batch_publish_spec_messages_array_objects(): assert messages[0]['name'] == 'event1' assert messages[0]['data'] == 'data1' assert messages[1]['name'] == 'event2' - assert messages[1]['data'] == '{"key":"value"}' + assert isinstance(messages[1]['data'], str) + assert json.loads(messages[1]['data']) == {'key': 'value'} assert messages[1]['encoding'] == 'json' # UTS: rest/unit/BPR2a/success-channel-name-0 -@deviation async def test_bpr2a_batch_publish_success_channel_name(): channel_name = f'test-BPR2a-{random_id()}' mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=respond_with(201, [success_result(channel_name, 'msg123', ['s1'])]), + on_request=respond_with(201, [batch_result(success_result(channel_name, 'msg123', ['s1']))]), ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) assert result.results[0].channel == channel_name # UTS: rest/unit/BPR2b/success-message-id-prefix-0 -@deviation async def test_bpr2b_batch_publish_success_message_id_prefix(): channel_name = f'test-BPR2b-{random_id()}' mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=respond_with(201, [success_result(channel_name, 'unique-id-prefix', ['s1', 's2'])]), + on_request=respond_with(201, [batch_result( + success_result(channel_name, 'unique-id-prefix', ['s1', 's2']), + )]), ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='e1', data='d1'), Message(name='e2', data='d2')], - }) + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='e1', data='d1'), Message(name='e2', data='d2')], + )) assert result.results[0].message_id == 'unique-id-prefix' # UTS: rest/unit/BPR2c/serials-array-0 -@deviation async def test_bpr2c_batch_publish_serials_array(): channel_name = f'test-BPR2c-{random_id()}' mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=respond_with(201, [success_result(channel_name, 'msg', ['serial1', 'serial2', 'serial3'])]), + on_request=respond_with(201, [batch_result( + success_result(channel_name, 'msg', ['serial1', 'serial2', 'serial3']), + )]), ) client = rest_client(mock_http) messages = [Message(name=f'e{index}', data=f'd{index}') for index in range(3)] - result = await client.batch_publish({'channels': [channel_name], 'messages': messages}) + result = await client.batch_publish(BatchPublishSpec(channels=[channel_name], messages=messages)) assert result.results[0].serials == ['serial1', 'serial2', 'serial3'] assert len(result.results[0].serials) == len(messages) # UTS: rest/unit/BPR2c/serials-null-conflated-0 -@deviation async def test_bpr2c_batch_publish_serials_null_conflated(): channel_name = f'test-BPR2c1-{random_id()}' mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=respond_with(201, [{ + on_request=respond_with(201, [batch_result({ 'channel': channel_name, 'messageId': 'msg', 'serials': ['serial1', None, 'serial3'], - }]), + })]), ) client = rest_client(mock_http) messages = [Message(name=f'e{index}', data=f'd{index}') for index in range(3)] - result = await client.batch_publish({'channels': [channel_name], 'messages': messages}) + result = await client.batch_publish(BatchPublishSpec(channels=[channel_name], messages=messages)) # BPR2c: a null serial marks a message discarded by a conflation rule assert result.results[0].serials == ['serial1', None, 'serial3'] # UTS: rest/unit/BPF2a/failure-channel-name-0 -@deviation async def test_bpf2a_batch_publish_failure_channel_name(): channel_name = f'test-BPF2a-{random_id()}' mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=respond_with(201, [{ + on_request=respond_with(201, [batch_result({ 'channel': channel_name, 'error': {'code': 40160, 'statusCode': 401, 'message': 'Not permitted'}, - }]), + })]), ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) assert result.results[0].channel == channel_name # UTS: rest/unit/BPF2b/failure-error-info-0 -@deviation async def test_bpf2b_batch_publish_failure_error_info(): channel_name = f'test-BPF2b-{random_id()}' mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=respond_with(201, [{ + on_request=respond_with(201, [batch_result({ 'channel': channel_name, 'error': { 'code': 40160, @@ -534,78 +544,78 @@ async def test_bpf2b_batch_publish_failure_error_info(): 'message': 'Channel operation not permitted', 'href': 'https://help.ably.io/error/40160', }, - }]), + })]), ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) error = result.results[0].error + assert isinstance(error, AblyException) assert error.code == 40160 assert error.status_code == 401 assert 'not permitted' in error.message # UTS: rest/unit/RSC22c/partial-success-mixed-results-0 -@deviation async def test_rsc22c_batch_publish_partial_success_mixed_results(): channel_name_allowed = f'test-BatchResult1-allowed-{random_id()}' channel_name_restricted = f'test-BatchResult1-restricted-{random_id()}' mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=respond_with(201, [ + on_request=respond_with(201, [batch_result( success_result(channel_name_allowed, 'msg1', ['s1']), { 'channel': channel_name_restricted, 'error': {'code': 40160, 'statusCode': 401, 'message': 'Not permitted'}, }, - ]), + )]), ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name_allowed, channel_name_restricted], - 'messages': [Message(name='event', data='hello')], - }) + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name_allowed, channel_name_restricted], + messages=[Message(name='event', data='hello')], + )) # NOTE: the spec writes result[0] / result[1]; BAR2c puts the per-channel results in # BatchResult.results, so they are read from there. assert len(result.results) == 2 + assert isinstance(result.results[0], BatchPublishSuccessResult) assert result.results[0].channel == channel_name_allowed assert result.results[0].message_id == 'msg1' - assert getattr(result.results[0], 'error', None) is None + assert isinstance(result.results[1], BatchPublishFailureResult) assert result.results[1].channel == channel_name_restricted assert result.results[1].error.code == 40160 # UTS: rest/unit/RSC22c/distinguish-success-failure-0 -@deviation async def test_rsc22c_batch_publish_distinguish_success_failure(): channel_name = f'test-BatchResult2-{random_id()}' failed_channel_name = f'test-BatchResult2-failed-{random_id()}' mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=respond_with(201, [ + on_request=respond_with(201, [batch_result( success_result(channel_name, 'msg1', ['s1']), { 'channel': failed_channel_name, 'error': {'code': 40160, 'statusCode': 401, 'message': 'Not permitted'}, }, - ]), + )]), ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name, failed_channel_name], - 'messages': [Message(name='event', data='hello')], - }) + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name, failed_channel_name], + messages=[Message(name='event', data='hello')], + )) for entry in result.results: has_success_fields = getattr(entry, 'message_id', None) is not None and \ @@ -615,7 +625,6 @@ async def test_rsc22c_batch_publish_distinguish_success_failure(): # UTS: rest/unit/RSC22/empty-channels-rejected-0 -@deviation async def test_rsc22_batch_publish_empty_channels_rejected(): mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), @@ -624,14 +633,13 @@ async def test_rsc22_batch_publish_empty_channels_rejected(): client = rest_client(mock_http) with pytest.raises(AblyException) as excinfo: - await client.batch_publish({'channels': [], 'messages': [Message(name='event', data='hello')]}) + await client.batch_publish(BatchPublishSpec(channels=[], messages=[Message(name='event', data='hello')])) # NOTE: the spec says only "the error indicates invalid request"; read as a 400. assert excinfo.value.status_code == 400 # UTS: rest/unit/RSC22/empty-messages-rejected-0 -@deviation async def test_rsc22_batch_publish_empty_messages_rejected(): channel_name = f'test-RSC22-Error2-{random_id()}' @@ -642,14 +650,13 @@ async def test_rsc22_batch_publish_empty_messages_rejected(): client = rest_client(mock_http) with pytest.raises(AblyException) as excinfo: - await client.batch_publish({'channels': [channel_name], 'messages': []}) + await client.batch_publish(BatchPublishSpec(channels=[channel_name], messages=[])) # NOTE: the spec says only "the error indicates invalid request"; read as a 400. assert excinfo.value.status_code == 400 # UTS: rest/unit/RSC22/server-error-propagated-0 -@deviation async def test_rsc22_batch_publish_server_error_propagated(): channel_name = f'test-RSC22-Error3-{random_id()}' @@ -662,17 +669,16 @@ async def test_rsc22_batch_publish_server_error_propagated(): client = rest_client(mock_http) with pytest.raises(AblyException) as excinfo: - await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) assert excinfo.value.code == 50000 assert excinfo.value.status_code == 500 # UTS: rest/unit/RSC22/auth-error-propagated-0 -@deviation async def test_rsc22_batch_publish_auth_error_propagated(): channel_name = f'test-RSC22-Error4-{random_id()}' @@ -685,17 +691,16 @@ async def test_rsc22_batch_publish_auth_error_propagated(): client = rest_client(mock_http) with pytest.raises(AblyException) as excinfo: - await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) assert excinfo.value.code == 40101 assert excinfo.value.status_code == 401 # UTS: rest/unit/RSC22/standard-headers-included-0 -@deviation async def test_rsc22_batch_publish_standard_headers_included(): channel_name = f'test-RSC22-Headers1-{random_id()}' @@ -706,10 +711,10 @@ async def test_rsc22_batch_publish_standard_headers_included(): ) client = rest_client(mock_http) - await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) assert len(captured_requests) == 1 request = captured_requests[0] @@ -726,7 +731,6 @@ async def test_rsc22_batch_publish_standard_headers_included(): # UTS: rest/unit/RSC22/request-id-included-0 -@deviation async def test_rsc22_batch_publish_request_id_included(): channel_name = f'test-RSC22-Headers2-{random_id()}' @@ -737,10 +741,10 @@ async def test_rsc22_batch_publish_request_id_included(): ) client = rest_client(mock_http, add_request_ids=True) - await client.batch_publish({ - 'channels': [channel_name], - 'messages': [Message(name='event', data='hello')], - }) + await client.batch_publish(BatchPublishSpec( + channels=[channel_name], + messages=[Message(name='event', data='hello')], + )) assert len(captured_requests) == 1 request_id = captured_requests[0].url.query_params['request_id'] @@ -751,21 +755,20 @@ async def test_rsc22_batch_publish_request_id_included(): # UTS: rest/unit/RSC22/multiple-messages-per-channel-0 -@deviation async def test_rsc22_batch_publish_multiple_messages_per_channel(): channel_name = f'test-RSC22-Batch1-{random_id()}' captured_requests = [] mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=capture_and_respond(captured_requests, body=[ + on_request=capture_and_respond(captured_requests, body=[batch_result( success_result(channel_name, 'msg', [f's{index}' for index in range(100)]), - ]), + )]), ) client = rest_client(mock_http) messages = [Message(name=f'event-{index}', data=f'data-{index}') for index in range(100)] - result = await client.batch_publish({'channels': [channel_name], 'messages': messages}) + result = await client.batch_publish(BatchPublishSpec(channels=[channel_name], messages=messages)) assert len(captured_requests) == 1 sent_messages = msgpack.unpackb(captured_requests[0].body)['messages'] @@ -777,7 +780,6 @@ async def test_rsc22_batch_publish_multiple_messages_per_channel(): # UTS: rest/unit/RSC22/multiple-channels-multiple-messages-0 -@deviation async def test_rsc22_batch_publish_multiple_channels_multiple_messages(): channel_name_1 = f'test-RSC22-Batch2-a-{random_id()}' channel_name_2 = f'test-RSC22-Batch2-b-{random_id()}' @@ -786,22 +788,22 @@ async def test_rsc22_batch_publish_multiple_channels_multiple_messages(): captured_requests = [] mock_http = MockHttpClient( on_connection_attempt=lambda conn: conn.respond_with_success(), - on_request=capture_and_respond(captured_requests, body=[ + on_request=capture_and_respond(captured_requests, body=[batch_result( success_result(channel_name_1, 'msg1', ['s1', 's2', 's3']), success_result(channel_name_2, 'msg2', ['s4', 's5', 's6']), success_result(channel_name_3, 'msg3', ['s7', 's8', 's9']), - ]), + )]), ) client = rest_client(mock_http) - result = await client.batch_publish({ - 'channels': [channel_name_1, channel_name_2, channel_name_3], - 'messages': [ + result = await client.batch_publish(BatchPublishSpec( + channels=[channel_name_1, channel_name_2, channel_name_3], + messages=[ Message(name='msg1', data='d1'), Message(name='msg2', data='d2'), Message(name='msg3', data='d3'), ], - }) + )) assert len(captured_requests) == 1 body = msgpack.unpackb(captured_requests[0].body) diff --git a/test/uts/rest/unit/channel/annotations_test.py b/test/uts/rest/unit/channel/annotations_test.py index 1dd1ab57..c3d39264 100644 --- a/test/uts/rest/unit/channel/annotations_test.py +++ b/test/uts/rest/unit/channel/annotations_test.py @@ -15,10 +15,10 @@ import msgpack import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.rest.annotations import RestAnnotations -from ably.types.annotation import Annotation, AnnotationAction -from ably.util.exceptions import AblyException +from ably.pubsub.http.annotations import HttpAnnotations +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient @@ -53,7 +53,7 @@ async def test_rsl10_annotations_attribute_type(): client = rest_client(mock_http) channel = client.channels.get('test-RSL10') - assert isinstance(channel.annotations, RestAnnotations) + assert isinstance(channel.annotations, HttpAnnotations) # UTS: rest/unit/RSAN1c6/publish-post-annotation-create-0 diff --git a/test/uts/rest/unit/channel/get_message_test.py b/test/uts/rest/unit/channel/get_message_test.py index 44838b3e..8f3f1571 100644 --- a/test/uts/rest/unit/channel/get_message_test.py +++ b/test/uts/rest/unit/channel/get_message_test.py @@ -7,8 +7,8 @@ import pytest -from ably.types.message import Message -from ably.util.exceptions import AblyException +from ably.pubsub.types.message import Message +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/channel/history_test.py b/test/uts/rest/unit/channel/history_test.py index c2fe1047..cfae1d4b 100644 --- a/test/uts/rest/unit/channel/history_test.py +++ b/test/uts/rest/unit/channel/history_test.py @@ -5,8 +5,8 @@ import uuid -from ably.http.paginatedresult import PaginatedResult -from ably.types.message import Message +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.message import Message from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/channel/idempotency_test.py b/test/uts/rest/unit/channel/idempotency_test.py index cb747579..35d8c5c5 100644 --- a/test/uts/rest/unit/channel/idempotency_test.py +++ b/test/uts/rest/unit/channel/idempotency_test.py @@ -8,7 +8,7 @@ import msgpack -from ably.types.message import Message +from ably.pubsub.types.message import Message from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import spec_error from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/channel/message_versions_test.py b/test/uts/rest/unit/channel/message_versions_test.py index 2a5c3001..b533c8da 100644 --- a/test/uts/rest/unit/channel/message_versions_test.py +++ b/test/uts/rest/unit/channel/message_versions_test.py @@ -5,8 +5,8 @@ import uuid -from ably.http.paginatedresult import PaginatedResult -from ably.types.message import Message, MessageAction +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.message import Message, MessageAction from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/channel/publish_result_test.py b/test/uts/rest/unit/channel/publish_result_test.py index f7120027..afc5f997 100644 --- a/test/uts/rest/unit/channel/publish_result_test.py +++ b/test/uts/rest/unit/channel/publish_result_test.py @@ -5,8 +5,8 @@ import uuid -from ably.types.message import Message -from ably.types.operations import PublishResult +from ably.pubsub.types.message import Message +from ably.pubsub.types.operations import PublishResult from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/channel/publish_test.py b/test/uts/rest/unit/channel/publish_test.py index f03fb397..7ebef9d4 100644 --- a/test/uts/rest/unit/channel/publish_test.py +++ b/test/uts/rest/unit/channel/publish_test.py @@ -8,8 +8,8 @@ import msgpack import pytest -from ably.types.message import Message -from ably.util.exceptions import AblyException +from ably.pubsub.types.message import Message +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation, spec_error from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/channel/rest_channel_attributes_test.py b/test/uts/rest/unit/channel/rest_channel_attributes_test.py index 7b2b2371..479ebd90 100644 --- a/test/uts/rest/unit/channel/rest_channel_attributes_test.py +++ b/test/uts/rest/unit/channel/rest_channel_attributes_test.py @@ -4,8 +4,8 @@ CHO2, CHO2a, CHM2, CHM2a, CHM2b, CHM2c, CHM2d, CHM2e, CHM2f, CHM2g, CHM2h """ -from ably.types.channeldetails import ChannelDetails -from ably.types.channeloptions import ChannelOptions +from ably.pubsub.types.channeldetails import ChannelDetails +from ably.pubsub.types.channeloptions import ChannelOptions from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/channel/update_delete_message_test.py b/test/uts/rest/unit/channel/update_delete_message_test.py index 153cc566..cb4de9c7 100644 --- a/test/uts/rest/unit/channel/update_delete_message_test.py +++ b/test/uts/rest/unit/channel/update_delete_message_test.py @@ -10,9 +10,9 @@ import msgpack import pytest -from ably.types.message import Message -from ably.types.operations import MessageOperation, UpdateDeleteResult -from ably.util.exceptions import AblyException +from ably.pubsub.types.message import Message +from ably.pubsub.types.operations import MessageOperation, UpdateDeleteResult +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/channels_collection_test.py b/test/uts/rest/unit/channels_collection_test.py index b7517fca..3a43f167 100644 --- a/test/uts/rest/unit/channels_collection_test.py +++ b/test/uts/rest/unit/channels_collection_test.py @@ -8,7 +8,7 @@ import uuid -from ably.rest.channel import Channel, Channels +from ably.pubsub.http.channel import Channel, Channels from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/encoding/message_encoding_test.py b/test/uts/rest/unit/encoding/message_encoding_test.py index 8a86ffc7..6d5d1957 100644 --- a/test/uts/rest/unit/encoding/message_encoding_test.py +++ b/test/uts/rest/unit/encoding/message_encoding_test.py @@ -10,7 +10,7 @@ import msgpack import pytest -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/encoding/msgpack_interop_test.py b/test/uts/rest/unit/encoding/msgpack_interop_test.py index ee681dec..f92d802e 100644 --- a/test/uts/rest/unit/encoding/msgpack_interop_test.py +++ b/test/uts/rest/unit/encoding/msgpack_interop_test.py @@ -10,7 +10,7 @@ import msgpack import pytest -from ably.types.message import Message +from ably.pubsub.types.message import Message FIXTURES_PATH = os.path.join( os.path.dirname(__file__), '..', '..', '..', '..', '..', diff --git a/test/uts/rest/unit/fallback_test.py b/test/uts/rest/unit/fallback_test.py index 31446900..3cc8eafd 100644 --- a/test/uts/rest/unit/fallback_test.py +++ b/test/uts/rest/unit/fallback_test.py @@ -9,7 +9,7 @@ import pytest -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/presence/rest_presence_test.py b/test/uts/rest/unit/presence/rest_presence_test.py index 09a98a3f..584949c8 100644 --- a/test/uts/rest/unit/presence/rest_presence_test.py +++ b/test/uts/rest/unit/presence/rest_presence_test.py @@ -11,10 +11,10 @@ import msgpack import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.types.presence import Presence, PresenceAction, PresenceMessage -from ably.util.crypto import CipherParams -from ably.util.exceptions import AblyException +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.presence import Presence, PresenceAction, PresenceMessage +from ably.pubsub.util.crypto import CipherParams +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation, spec_error from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/push/push_admin_publish_test.py b/test/uts/rest/unit/push/push_admin_publish_test.py index aaaec0c1..846baa70 100644 --- a/test/uts/rest/unit/push/push_admin_publish_test.py +++ b/test/uts/rest/unit/push/push_admin_publish_test.py @@ -6,8 +6,8 @@ import msgpack import pytest -from ably.rest.push import Push, PushAdmin, PushChannelSubscriptions, PushDeviceRegistrations -from ably.util.exceptions import AblyException +from ably.pubsub.http.push import Push, PushAdmin, PushChannelSubscriptions, PushDeviceRegistrations +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/push/push_channel_subscriptions_test.py b/test/uts/rest/unit/push/push_channel_subscriptions_test.py index 25ab045c..7826276f 100644 --- a/test/uts/rest/unit/push/push_channel_subscriptions_test.py +++ b/test/uts/rest/unit/push/push_channel_subscriptions_test.py @@ -6,9 +6,9 @@ import msgpack import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.types.channelsubscription import PushChannelSubscription -from ably.util.exceptions import AblyException +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.channelsubscription import PushChannelSubscription +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/push/push_channels_test.py b/test/uts/rest/unit/push/push_channels_test.py index b213f534..699dec11 100644 --- a/test/uts/rest/unit/push/push_channels_test.py +++ b/test/uts/rest/unit/push/push_channels_test.py @@ -7,16 +7,16 @@ ``push`` field on a channel) nor LocalDevice (RSH8), so every test in this file is gated behind RUN_DEVIATIONS. The assertions are written against the spec, using the names ably-python would use for these APIs: ``client.device``, -``ably.types.device.LocalDevice`` and ``channel.push.subscribe_device()`` and +``ably.pubsub.types.device.LocalDevice`` and ``channel.push.subscribe_device()`` and friends. Running them raises ImportError/AttributeError until the APIs land. """ import msgpack import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.types.channelsubscription import PushChannelSubscription -from ably.util.exceptions import AblyException +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.channelsubscription import PushChannelSubscription +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient @@ -30,7 +30,7 @@ def set_local_device(client, device_id=DEVICE_ID, device_identity_token=DEVICE_IDENTITY_TOKEN, client_id=CLIENT_ID): """Configure the client's local device, standing in for the spec's ``client.device = ...``.""" - from ably.types.device import LocalDevice + from ably.pubsub.types.device import LocalDevice client.device = LocalDevice( id=device_id, diff --git a/test/uts/rest/unit/push/push_device_registrations_test.py b/test/uts/rest/unit/push/push_device_registrations_test.py index ccf7e4a1..495ecc64 100644 --- a/test/uts/rest/unit/push/push_device_registrations_test.py +++ b/test/uts/rest/unit/push/push_device_registrations_test.py @@ -8,9 +8,9 @@ import msgpack import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.types.device import DeviceDetails -from ably.util.exceptions import AblyException +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.types.device import DeviceDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/rest_client_test.py b/test/uts/rest/unit/rest_client_test.py index 91731161..fd3454c5 100644 --- a/test/uts/rest/unit/rest_client_test.py +++ b/test/uts/rest/unit/rest_client_test.py @@ -8,9 +8,10 @@ import msgpack import pytest -from ably.rest.auth import Auth -from ably.util.exceptions import AblyException +from ably.pubsub.http.auth import Auth +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client +from test.uts.helpers.deviations import spec_error from test.uts.helpers.mock_http import MockHttpClient SERVER_TIME_MS = 1234567890000 @@ -56,6 +57,10 @@ def on_request(request): # UTS: rest/unit/RSC7d/ably-agent-header-format-0 +# SPEC ERROR RSC7d: the pattern `ably-[a-z]+/` rejects a hyphenated library name. +# RSC7d1 allows any product key, and ably-python reports itself as +# `ably-pubsub-python`. Recorded in deviations.md. Fix the specification first. +@spec_error async def test_rsc7d_ably_agent_header_format(): mock_http = MockHttpClient(on_connection_attempt=connect_successfully) mock_http.queue_response(200, TIME_RESPONSE) diff --git a/test/uts/rest/unit/stats_test.py b/test/uts/rest/unit/stats_test.py index 1e4bd328..0cfdaaa9 100644 --- a/test/uts/rest/unit/stats_test.py +++ b/test/uts/rest/unit/stats_test.py @@ -7,8 +7,8 @@ import pytest -from ably.http.paginatedresult import PaginatedResult -from ably.util.exceptions import AblyException +from ably.pubsub.request.paginatedresult import PaginatedResult +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/time_test.py b/test/uts/rest/unit/time_test.py index ad2d4b6e..b7504395 100644 --- a/test/uts/rest/unit/time_test.py +++ b/test/uts/rest/unit/time_test.py @@ -5,7 +5,7 @@ import pytest -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/types/error_types_test.py b/test/uts/rest/unit/types/error_types_test.py index ae6f0ce6..ef46afec 100644 --- a/test/uts/rest/unit/types/error_types_test.py +++ b/test/uts/rest/unit/types/error_types_test.py @@ -9,7 +9,7 @@ import pytest -from ably.util.exceptions import AblyException +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.deviations import deviation diff --git a/test/uts/rest/unit/types/message_types_test.py b/test/uts/rest/unit/types/message_types_test.py index df2ae6ff..b0db98ac 100644 --- a/test/uts/rest/unit/types/message_types_test.py +++ b/test/uts/rest/unit/types/message_types_test.py @@ -5,7 +5,7 @@ import pytest -from ably.types.message import Message +from ably.pubsub.types.message import Message # `fromEncoded` is spelled `from_encoded` here, and "no encoding" is rendered as the empty # string rather than null, since EncodeDataMixin joins an empty list of transforms. diff --git a/test/uts/rest/unit/types/mutable_message_types_test.py b/test/uts/rest/unit/types/mutable_message_types_test.py index 180e117f..72ee6efb 100644 --- a/test/uts/rest/unit/types/mutable_message_types_test.py +++ b/test/uts/rest/unit/types/mutable_message_types_test.py @@ -4,9 +4,9 @@ MOP2a, MOP2b, MOP2c, UDR1, UDR2, UDR2a, TAN1, TAN2, TAN2a-TAN2l """ -from ably.types.annotation import Annotation, AnnotationAction -from ably.types.message import Message, MessageAction, MessageAnnotations, MessageVersion -from ably.types.operations import MessageOperation, UpdateDeleteResult +from ably.pubsub.types.annotation import Annotation, AnnotationAction +from ably.pubsub.types.message import Message, MessageAction, MessageAnnotations, MessageVersion +from ably.pubsub.types.operations import MessageOperation, UpdateDeleteResult # `fromJson` is spelled `from_encoded` for wire payloads and `from_dict` for plain records, # and `toJson` is spelled `as_dict`. diff --git a/test/uts/rest/unit/types/options_types_test.py b/test/uts/rest/unit/types/options_types_test.py index fb443069..bc5848cd 100644 --- a/test/uts/rest/unit/types/options_types_test.py +++ b/test/uts/rest/unit/types/options_types_test.py @@ -5,11 +5,11 @@ import pytest -from ably import AblyRest -from ably.types.authoptions import AuthOptions -from ably.types.options import Options -from ably.types.tokendetails import TokenDetails -from ably.util.exceptions import AblyException +from ably.pubsub.server import create_http_client +from ably.pubsub.types.authoptions import AuthOptions +from ably.pubsub.types.options import Options +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.deviations import deviation from test.uts.helpers.mock_http import MockHttpClient @@ -191,5 +191,5 @@ async def auth_callback(params): # The missing-credentials check is a constructor argument check, so it raises ValueError # rather than an AblyException with pytest.raises(ValueError) as value_error: - AblyRest() + create_http_client() assert 'key' in str(value_error.value) diff --git a/test/uts/rest/unit/types/paginated_result_test.py b/test/uts/rest/unit/types/paginated_result_test.py index a18525e4..22f0f1bb 100644 --- a/test/uts/rest/unit/types/paginated_result_test.py +++ b/test/uts/rest/unit/types/paginated_result_test.py @@ -5,9 +5,9 @@ import pytest -from ably.types.message import Message -from ably.types.presence import PresenceMessage -from ably.util.exceptions import AblyException +from ably.pubsub.types.message import Message +from ably.pubsub.types.presence import PresenceMessage +from ably.pubsub.util.exceptions import AblyException from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/test/uts/rest/unit/types/presence_message_types_test.py b/test/uts/rest/unit/types/presence_message_types_test.py index be0f044f..7c6b2b7c 100644 --- a/test/uts/rest/unit/types/presence_message_types_test.py +++ b/test/uts/rest/unit/types/presence_message_types_test.py @@ -5,7 +5,7 @@ from datetime import datetime, timedelta -from ably.types.presence import PresenceAction, PresenceMessage +from ably.pubsub.types.presence import PresenceAction, PresenceMessage from test.uts.helpers.deviations import deviation, spec_error diff --git a/test/uts/rest/unit/types/token_types_test.py b/test/uts/rest/unit/types/token_types_test.py index bc8764af..243291b3 100644 --- a/test/uts/rest/unit/types/token_types_test.py +++ b/test/uts/rest/unit/types/token_types_test.py @@ -10,8 +10,8 @@ import json -from ably.types.tokendetails import TokenDetails -from ably.types.tokenrequest import TokenRequest +from ably.pubsub.types.tokendetails import TokenDetails +from ably.pubsub.types.tokenrequest import TokenRequest from test.uts.helpers.client import rest_client from test.uts.helpers.mock_http import MockHttpClient diff --git a/uv.lock b/uv.lock index 29ccca45..b8f70971 100644 --- a/uv.lock +++ b/uv.lock @@ -1,29 +1,24 @@ version = 1 revision = 3 -requires-python = ">=3.7" +requires-python = ">=3.8" resolution-markers = [ "python_full_version >= '3.10'", "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", - "python_full_version < '3.8'", + "python_full_version < '3.9'", ] [[package]] -name = "ably" +name = "ably-pubsub-server" version = "3.1.4" source = { editable = "." } dependencies = [ { name = "h2", version = "4.1.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "h2", version = "4.3.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, - { name = "httpx", version = "0.24.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "httpx", version = "0.28.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, - { name = "msgpack", version = "1.0.5", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "msgpack", version = "1.1.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "httpx" }, + { name = "msgpack", version = "1.1.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "msgpack", version = "1.1.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, - { name = "pyee", version = "9.1.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "pyee", version = "13.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, - { name = "websockets", version = "11.0.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "websockets", version = "13.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "pyee" }, + { name = "websockets", version = "13.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "websockets", version = "15.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, ] @@ -32,20 +27,16 @@ crypto = [ { name = "pycryptodome" }, ] dev = [ - { name = "async-case", marker = "python_full_version < '3.8'" }, { name = "importlib-metadata" }, { name = "mock" }, { name = "pytest" }, - { name = "pytest-asyncio", version = "0.21.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "pytest-asyncio", version = "0.23.8", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, + { name = "pytest-asyncio" }, { name = "pytest-cov" }, { name = "pytest-timeout" }, { name = "pytest-xdist" }, - { name = "respx", version = "0.20.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "respx", version = "0.22.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, + { name = "respx" }, { name = "ruff" }, - { name = "tokenize-rt", version = "5.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "tokenize-rt", version = "6.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "tokenize-rt", version = "6.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "tokenize-rt", version = "6.2.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, { name = "vcdiff-decoder" }, ] @@ -87,36 +78,18 @@ requires-dist = [ ] provides-extras = ["oldcrypto", "crypto", "vcdiff", "dev"] -[[package]] -name = "anyio" -version = "3.7.1" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -dependencies = [ - { name = "exceptiongroup", marker = "python_full_version < '3.8'" }, - { name = "idna", version = "3.10", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "sniffio", marker = "python_full_version < '3.8'" }, - { name = "typing-extensions", version = "4.7.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/28/99/2dfd53fd55ce9838e6ff2d4dac20ce58263798bd1a0dbe18b3a9af3fcfce/anyio-3.7.1.tar.gz", hash = "sha256:44a3c9aba0f5defa43261a8b3efb97891f2bd7d804e0e1f56419befa1adfc780", size = 142927, upload-time = "2023-07-05T16:45:02.294Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl", hash = "sha256:91dee416e570e92c64041bd18b900d1d6fa78dff7048769ce5ac5ddad004fbb5", size = 80896, upload-time = "2023-07-05T16:44:59.805Z" }, -] - [[package]] name = "anyio" version = "4.5.2" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] dependencies = [ - { name = "exceptiongroup", marker = "python_full_version == '3.8.*'" }, - { name = "idna", version = "3.11", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, - { name = "sniffio", marker = "python_full_version == '3.8.*'" }, - { name = "typing-extensions", version = "4.13.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "exceptiongroup", marker = "python_full_version < '3.9'" }, + { name = "idna", marker = "python_full_version < '3.9'" }, + { name = "sniffio", marker = "python_full_version < '3.9'" }, + { name = "typing-extensions", version = "4.13.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/4d/f9/9a7ce600ebe7804daf90d4d48b1c0510a4561ddce43a596be46676f82343/anyio-4.5.2.tar.gz", hash = "sha256:23009af4ed04ce05991845451e11ef02fc7c5ed29179ac9a420e5ad0ac7ddc5b", size = 171293, upload-time = "2024-10-13T22:18:03.307Z" } wheels = [ @@ -133,7 +106,7 @@ resolution-markers = [ ] dependencies = [ { name = "exceptiongroup", marker = "python_full_version >= '3.9' and python_full_version < '3.11'" }, - { name = "idna", version = "3.11", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, + { name = "idna", marker = "python_full_version >= '3.9'" }, { name = "sniffio", marker = "python_full_version >= '3.9'" }, { name = "typing-extensions", version = "4.15.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9' and python_full_version < '3.13'" }, ] @@ -142,12 +115,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/15/b3/9b1a8074496371342ec1e796a96f99c82c945a339cd81a8e73de28b4cf9e/anyio-4.11.0-py3-none-any.whl", hash = "sha256:0287e96f4d26d4149305414d4e3bc32f0dcd0862365a4bddea19d7a1ec38c4fc", size = 109097, upload-time = "2025-09-23T09:19:10.601Z" }, ] -[[package]] -name = "async-case" -version = "10.1.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/c8/09/87f2a23f5696ac6deb2fff92421f8af46226ea2410d101b453d5aa63e53a/async_case-10.1.0.tar.gz", hash = "sha256:b819f68c78f6c640ab1101ecf69fac189402b490901fa2abc314c48edab5d3da", size = 3668, upload-time = "2022-03-15T21:56:16.795Z" } - [[package]] name = "certifi" version = "2026.1.4" @@ -166,82 +133,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, ] -[[package]] -name = "coverage" -version = "7.2.7" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/45/8b/421f30467e69ac0e414214856798d4bc32da1336df745e49e49ae5c1e2a8/coverage-7.2.7.tar.gz", hash = "sha256:924d94291ca674905fe9481f12294eb11f2d3d3fd1adb20314ba89e94f44ed59", size = 762575, upload-time = "2023-05-29T20:08:50.273Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/01/24/be01e62a7bce89bcffe04729c540382caa5a06bee45ae42136c93e2499f5/coverage-7.2.7-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d39b5b4f2a66ccae8b7263ac3c8170994b65266797fb96cbbfd3fb5b23921db8", size = 200724, upload-time = "2023-05-29T20:07:03.422Z" }, - { url = "https://files.pythonhosted.org/packages/3d/80/7060a445e1d2c9744b683dc935248613355657809d6c6b2716cdf4ca4766/coverage-7.2.7-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:6d040ef7c9859bb11dfeb056ff5b3872436e3b5e401817d87a31e1750b9ae2fb", size = 201024, upload-time = "2023-05-29T20:07:05.694Z" }, - { url = "https://files.pythonhosted.org/packages/b8/9d/926fce7e03dbfc653104c2d981c0fa71f0572a9ebd344d24c573bd6f7c4f/coverage-7.2.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ba90a9563ba44a72fda2e85302c3abc71c5589cea608ca16c22b9804262aaeb6", size = 229528, upload-time = "2023-05-29T20:07:07.307Z" }, - { url = "https://files.pythonhosted.org/packages/d1/3a/67f5d18f911abf96857f6f7e4df37ca840e38179e2cc9ab6c0b9c3380f19/coverage-7.2.7-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e7d9405291c6928619403db1d10bd07888888ec1abcbd9748fdaa971d7d661b2", size = 227842, upload-time = "2023-05-29T20:07:09.331Z" }, - { url = "https://files.pythonhosted.org/packages/b4/bd/1b2331e3a04f4cc9b7b332b1dd0f3a1261dfc4114f8479bebfcc2afee9e8/coverage-7.2.7-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:31563e97dae5598556600466ad9beea39fb04e0229e61c12eaa206e0aa202063", size = 228717, upload-time = "2023-05-29T20:07:11.38Z" }, - { url = "https://files.pythonhosted.org/packages/2b/86/3dbf9be43f8bf6a5ca28790a713e18902b2d884bc5fa9512823a81dff601/coverage-7.2.7-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:ebba1cd308ef115925421d3e6a586e655ca5a77b5bf41e02eb0e4562a111f2d1", size = 234632, upload-time = "2023-05-29T20:07:13.376Z" }, - { url = "https://files.pythonhosted.org/packages/91/e8/469ed808a782b9e8305a08bad8c6fa5f8e73e093bda6546c5aec68275bff/coverage-7.2.7-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:cb017fd1b2603ef59e374ba2063f593abe0fc45f2ad9abdde5b4d83bd922a353", size = 232875, upload-time = "2023-05-29T20:07:15.093Z" }, - { url = "https://files.pythonhosted.org/packages/29/8f/4fad1c2ba98104425009efd7eaa19af9a7c797e92d40cd2ec026fa1f58cb/coverage-7.2.7-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:d62a5c7dad11015c66fbb9d881bc4caa5b12f16292f857842d9d1871595f4495", size = 234094, upload-time = "2023-05-29T20:07:17.013Z" }, - { url = "https://files.pythonhosted.org/packages/94/4e/d4e46a214ae857be3d7dc5de248ba43765f60daeb1ab077cb6c1536c7fba/coverage-7.2.7-cp310-cp310-win32.whl", hash = "sha256:ee57190f24fba796e36bb6d3aa8a8783c643d8fa9760c89f7a98ab5455fbf818", size = 203184, upload-time = "2023-05-29T20:07:18.69Z" }, - { url = "https://files.pythonhosted.org/packages/1f/e9/d6730247d8dec2a3dddc520ebe11e2e860f0f98cee3639e23de6cf920255/coverage-7.2.7-cp310-cp310-win_amd64.whl", hash = "sha256:f75f7168ab25dd93110c8a8117a22450c19976afbc44234cbf71481094c1b850", size = 204096, upload-time = "2023-05-29T20:07:20.153Z" }, - { url = "https://files.pythonhosted.org/packages/c6/fa/529f55c9a1029c840bcc9109d5a15ff00478b7ff550a1ae361f8745f8ad5/coverage-7.2.7-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:06a9a2be0b5b576c3f18f1a241f0473575c4a26021b52b2a85263a00f034d51f", size = 200895, upload-time = "2023-05-29T20:07:21.963Z" }, - { url = "https://files.pythonhosted.org/packages/67/d7/cd8fe689b5743fffac516597a1222834c42b80686b99f5b44ef43ccc2a43/coverage-7.2.7-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:5baa06420f837184130752b7c5ea0808762083bf3487b5038d68b012e5937dbe", size = 201120, upload-time = "2023-05-29T20:07:23.765Z" }, - { url = "https://files.pythonhosted.org/packages/8c/95/16eed713202406ca0a37f8ac259bbf144c9d24f9b8097a8e6ead61da2dbb/coverage-7.2.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fdec9e8cbf13a5bf63290fc6013d216a4c7232efb51548594ca3631a7f13c3a3", size = 233178, upload-time = "2023-05-29T20:07:25.281Z" }, - { url = "https://files.pythonhosted.org/packages/c1/49/4d487e2ad5d54ed82ac1101e467e8994c09d6123c91b2a962145f3d262c2/coverage-7.2.7-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:52edc1a60c0d34afa421c9c37078817b2e67a392cab17d97283b64c5833f427f", size = 230754, upload-time = "2023-05-29T20:07:27.044Z" }, - { url = "https://files.pythonhosted.org/packages/a7/cd/3ce94ad9d407a052dc2a74fbeb1c7947f442155b28264eb467ee78dea812/coverage-7.2.7-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:63426706118b7f5cf6bb6c895dc215d8a418d5952544042c8a2d9fe87fcf09cb", size = 232558, upload-time = "2023-05-29T20:07:28.743Z" }, - { url = "https://files.pythonhosted.org/packages/8f/a8/12cc7b261f3082cc299ab61f677f7e48d93e35ca5c3c2f7241ed5525ccea/coverage-7.2.7-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:afb17f84d56068a7c29f5fa37bfd38d5aba69e3304af08ee94da8ed5b0865833", size = 241509, upload-time = "2023-05-29T20:07:30.434Z" }, - { url = "https://files.pythonhosted.org/packages/04/fa/43b55101f75a5e9115259e8be70ff9279921cb6b17f04c34a5702ff9b1f7/coverage-7.2.7-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:48c19d2159d433ccc99e729ceae7d5293fbffa0bdb94952d3579983d1c8c9d97", size = 239924, upload-time = "2023-05-29T20:07:32.065Z" }, - { url = "https://files.pythonhosted.org/packages/68/5f/d2bd0f02aa3c3e0311986e625ccf97fdc511b52f4f1a063e4f37b624772f/coverage-7.2.7-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:0e1f928eaf5469c11e886fe0885ad2bf1ec606434e79842a879277895a50942a", size = 240977, upload-time = "2023-05-29T20:07:34.184Z" }, - { url = "https://files.pythonhosted.org/packages/ba/92/69c0722882643df4257ecc5437b83f4c17ba9e67f15dc6b77bad89b6982e/coverage-7.2.7-cp311-cp311-win32.whl", hash = "sha256:33d6d3ea29d5b3a1a632b3c4e4f4ecae24ef170b0b9ee493883f2df10039959a", size = 203168, upload-time = "2023-05-29T20:07:35.869Z" }, - { url = "https://files.pythonhosted.org/packages/b1/96/c12ed0dfd4ec587f3739f53eb677b9007853fd486ccb0e7d5512a27bab2e/coverage-7.2.7-cp311-cp311-win_amd64.whl", hash = "sha256:5b7540161790b2f28143191f5f8ec02fb132660ff175b7747b95dcb77ac26562", size = 204185, upload-time = "2023-05-29T20:07:37.39Z" }, - { url = "https://files.pythonhosted.org/packages/ff/d5/52fa1891d1802ab2e1b346d37d349cb41cdd4fd03f724ebbf94e80577687/coverage-7.2.7-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:f2f67fe12b22cd130d34d0ef79206061bfb5eda52feb6ce0dba0644e20a03cf4", size = 201020, upload-time = "2023-05-29T20:07:38.724Z" }, - { url = "https://files.pythonhosted.org/packages/24/df/6765898d54ea20e3197a26d26bb65b084deefadd77ce7de946b9c96dfdc5/coverage-7.2.7-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a342242fe22407f3c17f4b499276a02b01e80f861f1682ad1d95b04018e0c0d4", size = 233994, upload-time = "2023-05-29T20:07:40.274Z" }, - { url = "https://files.pythonhosted.org/packages/15/81/b108a60bc758b448c151e5abceed027ed77a9523ecbc6b8a390938301841/coverage-7.2.7-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:171717c7cb6b453aebac9a2ef603699da237f341b38eebfee9be75d27dc38e01", size = 231358, upload-time = "2023-05-29T20:07:41.998Z" }, - { url = "https://files.pythonhosted.org/packages/61/90/c76b9462f39897ebd8714faf21bc985b65c4e1ea6dff428ea9dc711ed0dd/coverage-7.2.7-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:49969a9f7ffa086d973d91cec8d2e31080436ef0fb4a359cae927e742abfaaa6", size = 233316, upload-time = "2023-05-29T20:07:43.539Z" }, - { url = "https://files.pythonhosted.org/packages/04/d6/8cba3bf346e8b1a4fb3f084df7d8cea25a6b6c56aaca1f2e53829be17e9e/coverage-7.2.7-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:b46517c02ccd08092f4fa99f24c3b83d8f92f739b4657b0f146246a0ca6a831d", size = 240159, upload-time = "2023-05-29T20:07:44.982Z" }, - { url = "https://files.pythonhosted.org/packages/6e/ea/4a252dc77ca0605b23d477729d139915e753ee89e4c9507630e12ad64a80/coverage-7.2.7-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:a3d33a6b3eae87ceaefa91ffdc130b5e8536182cd6dfdbfc1aa56b46ff8c86de", size = 238127, upload-time = "2023-05-29T20:07:46.522Z" }, - { url = "https://files.pythonhosted.org/packages/9f/5c/d9760ac497c41f9c4841f5972d0edf05d50cad7814e86ee7d133ec4a0ac8/coverage-7.2.7-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:976b9c42fb2a43ebf304fa7d4a310e5f16cc99992f33eced91ef6f908bd8f33d", size = 239833, upload-time = "2023-05-29T20:07:47.992Z" }, - { url = "https://files.pythonhosted.org/packages/69/8c/26a95b08059db1cbb01e4b0e6d40f2e9debb628c6ca86b78f625ceaf9bab/coverage-7.2.7-cp312-cp312-win32.whl", hash = "sha256:8de8bb0e5ad103888d65abef8bca41ab93721647590a3f740100cd65c3b00511", size = 203463, upload-time = "2023-05-29T20:07:49.939Z" }, - { url = "https://files.pythonhosted.org/packages/b7/00/14b00a0748e9eda26e97be07a63cc911108844004687321ddcc213be956c/coverage-7.2.7-cp312-cp312-win_amd64.whl", hash = "sha256:9e31cb64d7de6b6f09702bb27c02d1904b3aebfca610c12772452c4e6c21a0d3", size = 204347, upload-time = "2023-05-29T20:07:51.909Z" }, - { url = "https://files.pythonhosted.org/packages/80/d7/67937c80b8fd4c909fdac29292bc8b35d9505312cff6bcab41c53c5b1df6/coverage-7.2.7-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:58c2ccc2f00ecb51253cbe5d8d7122a34590fac9646a960d1430d5b15321d95f", size = 200580, upload-time = "2023-05-29T20:07:54.076Z" }, - { url = "https://files.pythonhosted.org/packages/7a/05/084864fa4bbf8106f44fb72a56e67e0cd372d3bf9d893be818338c81af5d/coverage-7.2.7-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d22656368f0e6189e24722214ed8d66b8022db19d182927b9a248a2a8a2f67eb", size = 226237, upload-time = "2023-05-29T20:07:56.28Z" }, - { url = "https://files.pythonhosted.org/packages/67/a2/6fa66a50e6e894286d79a3564f42bd54a9bd27049dc0a63b26d9924f0aa3/coverage-7.2.7-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a895fcc7b15c3fc72beb43cdcbdf0ddb7d2ebc959edac9cef390b0d14f39f8a9", size = 224256, upload-time = "2023-05-29T20:07:58.189Z" }, - { url = "https://files.pythonhosted.org/packages/e2/c0/73f139794c742840b9ab88e2e17fe14a3d4668a166ff95d812ac66c0829d/coverage-7.2.7-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e84606b74eb7de6ff581a7915e2dab7a28a0517fbe1c9239eb227e1354064dcd", size = 225550, upload-time = "2023-05-29T20:08:00.383Z" }, - { url = "https://files.pythonhosted.org/packages/03/ec/6f30b4e0c96ce03b0e64aec46b4af2a8c49b70d1b5d0d69577add757b946/coverage-7.2.7-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:0a5f9e1dbd7fbe30196578ca36f3fba75376fb99888c395c5880b355e2875f8a", size = 232440, upload-time = "2023-05-29T20:08:02.495Z" }, - { url = "https://files.pythonhosted.org/packages/22/c1/2f6c1b6f01a0996c9e067a9c780e1824351dbe17faae54388a4477e6d86f/coverage-7.2.7-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:419bfd2caae268623dd469eff96d510a920c90928b60f2073d79f8fe2bbc5959", size = 230897, upload-time = "2023-05-29T20:08:04.382Z" }, - { url = "https://files.pythonhosted.org/packages/8d/d6/53e999ec1bf7498ca4bc5f3b8227eb61db39068d2de5dcc359dec5601b5a/coverage-7.2.7-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:2aee274c46590717f38ae5e4650988d1af340fe06167546cc32fe2f58ed05b02", size = 232024, upload-time = "2023-05-29T20:08:06.031Z" }, - { url = "https://files.pythonhosted.org/packages/e9/40/383305500d24122dbed73e505a4d6828f8f3356d1f68ab6d32c781754b81/coverage-7.2.7-cp37-cp37m-win32.whl", hash = "sha256:61b9a528fb348373c433e8966535074b802c7a5d7f23c4f421e6c6e2f1697a6f", size = 203293, upload-time = "2023-05-29T20:08:07.598Z" }, - { url = "https://files.pythonhosted.org/packages/0e/bc/7e3a31534fabb043269f14fb64e2bb2733f85d4cf39e5bbc71357c57553a/coverage-7.2.7-cp37-cp37m-win_amd64.whl", hash = "sha256:b1c546aca0ca4d028901d825015dc8e4d56aac4b541877690eb76490f1dc8ed0", size = 204040, upload-time = "2023-05-29T20:08:09.919Z" }, - { url = "https://files.pythonhosted.org/packages/c6/fc/be19131010930a6cf271da48202c8cc1d3f971f68c02fb2d3a78247f43dc/coverage-7.2.7-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:54b896376ab563bd38453cecb813c295cf347cf5906e8b41d340b0321a5433e5", size = 200689, upload-time = "2023-05-29T20:08:11.594Z" }, - { url = "https://files.pythonhosted.org/packages/28/d7/9a8de57d87f4bbc6f9a6a5ded1eaac88a89bf71369bb935dac3c0cf2893e/coverage-7.2.7-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:3d376df58cc111dc8e21e3b6e24606b5bb5dee6024f46a5abca99124b2229ef5", size = 200986, upload-time = "2023-05-29T20:08:13.228Z" }, - { url = "https://files.pythonhosted.org/packages/c8/e4/e6182e4697665fb594a7f4e4f27cb3a4dd00c2e3d35c5c706765de8c7866/coverage-7.2.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5e330fc79bd7207e46c7d7fd2bb4af2963f5f635703925543a70b99574b0fea9", size = 230648, upload-time = "2023-05-29T20:08:15.11Z" }, - { url = "https://files.pythonhosted.org/packages/7b/e3/f552d5871943f747165b92a924055c5d6daa164ae659a13f9018e22f3990/coverage-7.2.7-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1e9d683426464e4a252bf70c3498756055016f99ddaec3774bf368e76bbe02b6", size = 228511, upload-time = "2023-05-29T20:08:16.877Z" }, - { url = "https://files.pythonhosted.org/packages/44/55/49f65ccdd4dfd6d5528e966b28c37caec64170c725af32ab312889d2f857/coverage-7.2.7-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8d13c64ee2d33eccf7437961b6ea7ad8673e2be040b4f7fd4fd4d4d28d9ccb1e", size = 229852, upload-time = "2023-05-29T20:08:18.47Z" }, - { url = "https://files.pythonhosted.org/packages/0d/31/340428c238eb506feb96d4fb5c9ea614db1149517f22cc7ab8c6035ef6d9/coverage-7.2.7-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:b7aa5f8a41217360e600da646004f878250a0d6738bcdc11a0a39928d7dc2050", size = 235578, upload-time = "2023-05-29T20:08:20.298Z" }, - { url = "https://files.pythonhosted.org/packages/dd/ce/97c1dd6592c908425622fe7f31c017d11cf0421729b09101d4de75bcadc8/coverage-7.2.7-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:8fa03bce9bfbeeef9f3b160a8bed39a221d82308b4152b27d82d8daa7041fee5", size = 234079, upload-time = "2023-05-29T20:08:22.365Z" }, - { url = "https://files.pythonhosted.org/packages/de/a3/5a98dc9e239d0dc5f243ef5053d5b1bdcaa1dee27a691dfc12befeccf878/coverage-7.2.7-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:245167dd26180ab4c91d5e1496a30be4cd721a5cf2abf52974f965f10f11419f", size = 234991, upload-time = "2023-05-29T20:08:24.974Z" }, - { url = "https://files.pythonhosted.org/packages/4a/fb/78986d3022e5ccf2d4370bc43a5fef8374f092b3c21d32499dee8e30b7b6/coverage-7.2.7-cp38-cp38-win32.whl", hash = "sha256:d2c2db7fd82e9b72937969bceac4d6ca89660db0a0967614ce2481e81a0b771e", size = 203160, upload-time = "2023-05-29T20:08:26.701Z" }, - { url = "https://files.pythonhosted.org/packages/c3/1c/6b3c9c363fb1433c79128e0d692863deb761b1b78162494abb9e5c328bc0/coverage-7.2.7-cp38-cp38-win_amd64.whl", hash = "sha256:2e07b54284e381531c87f785f613b833569c14ecacdcb85d56b25c4622c16c3c", size = 204085, upload-time = "2023-05-29T20:08:28.146Z" }, - { url = "https://files.pythonhosted.org/packages/88/da/495944ebf0ad246235a6bd523810d9f81981f9b81c6059ba1f56e943abe0/coverage-7.2.7-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:537891ae8ce59ef63d0123f7ac9e2ae0fc8b72c7ccbe5296fec45fd68967b6c9", size = 200725, upload-time = "2023-05-29T20:08:29.851Z" }, - { url = "https://files.pythonhosted.org/packages/ca/0c/3dfeeb1006c44b911ee0ed915350db30325d01808525ae7cc8d57643a2ce/coverage-7.2.7-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:06fb182e69f33f6cd1d39a6c597294cff3143554b64b9825d1dc69d18cc2fff2", size = 201022, upload-time = "2023-05-29T20:08:31.429Z" }, - { url = "https://files.pythonhosted.org/packages/61/af/5964b8d7d9a5c767785644d9a5a63cacba9a9c45cc42ba06d25895ec87be/coverage-7.2.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:201e7389591af40950a6480bd9edfa8ed04346ff80002cec1a66cac4549c1ad7", size = 229102, upload-time = "2023-05-29T20:08:32.982Z" }, - { url = "https://files.pythonhosted.org/packages/d9/1d/cd467fceb62c371f9adb1d739c92a05d4e550246daa90412e711226bd320/coverage-7.2.7-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f6951407391b639504e3b3be51b7ba5f3528adbf1a8ac3302b687ecababf929e", size = 227441, upload-time = "2023-05-29T20:08:35.044Z" }, - { url = "https://files.pythonhosted.org/packages/fe/57/e4f8ad64d84ca9e759d783a052795f62a9f9111585e46068845b1cb52c2b/coverage-7.2.7-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6f48351d66575f535669306aa7d6d6f71bc43372473b54a832222803eb956fd1", size = 228265, upload-time = "2023-05-29T20:08:36.861Z" }, - { url = "https://files.pythonhosted.org/packages/88/8b/b0d9fe727acae907fa7f1c8194ccb6fe9d02e1c3e9001ecf74c741f86110/coverage-7.2.7-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:b29019c76039dc3c0fd815c41392a044ce555d9bcdd38b0fb60fb4cd8e475ba9", size = 234217, upload-time = "2023-05-29T20:08:38.837Z" }, - { url = "https://files.pythonhosted.org/packages/66/2e/c99fe1f6396d93551aa352c75410686e726cd4ea104479b9af1af22367ce/coverage-7.2.7-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:81c13a1fc7468c40f13420732805a4c38a105d89848b7c10af65a90beff25250", size = 232466, upload-time = "2023-05-29T20:08:40.768Z" }, - { url = "https://files.pythonhosted.org/packages/bb/e9/88747b40c8fb4a783b40222510ce6d66170217eb05d7f46462c36b4fa8cc/coverage-7.2.7-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:975d70ab7e3c80a3fe86001d8751f6778905ec723f5b110aed1e450da9d4b7f2", size = 233669, upload-time = "2023-05-29T20:08:42.944Z" }, - { url = "https://files.pythonhosted.org/packages/b1/d5/a8e276bc005e42114468d4fe03e0a9555786bc51cbfe0d20827a46c1565a/coverage-7.2.7-cp39-cp39-win32.whl", hash = "sha256:7ee7d9d4822c8acc74a5e26c50604dff824710bc8de424904c0982e25c39c6cb", size = 203199, upload-time = "2023-05-29T20:08:44.734Z" }, - { url = "https://files.pythonhosted.org/packages/a9/0c/4a848ae663b47f1195abcb09a951751dd61f80b503303b9b9d768e0fd321/coverage-7.2.7-cp39-cp39-win_amd64.whl", hash = "sha256:eb393e5ebc85245347950143969b241d08b52b88a3dc39479822e073a1a8eb27", size = 204109, upload-time = "2023-05-29T20:08:46.417Z" }, - { url = "https://files.pythonhosted.org/packages/67/fb/b3b1d7887e1ea25a9608b0776e480e4bbc303ca95a31fd585555ec4fff5a/coverage-7.2.7-pp37.pp38.pp39-none-any.whl", hash = "sha256:b7b4c971f05e6ae490fef852c218b0e79d4e52f79ef0c8475566584a8fb3e01d", size = 193207, upload-time = "2023-05-29T20:08:48.153Z" }, -] - [[package]] name = "coverage" version = "7.6.1" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/f7/08/7e37f82e4d1aead42a7443ff06a1e406aabf7302c4f00a546e4b320b994c/coverage-7.6.1.tar.gz", hash = "sha256:953510dfb7b12ab69d20135a0662397f077c59b1e6379a768e97c59d852ee51d", size = 798791, upload-time = "2024-08-04T19:45:30.9Z" } wheels = [ @@ -539,8 +436,7 @@ name = "exceptiongroup" version = "1.3.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", version = "4.7.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "typing-extensions", version = "4.13.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "typing-extensions", version = "4.13.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "typing-extensions", version = "4.15.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9' and python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/0b/9f/a65090624ecf468cdca03533906e7c69ed7588582240cfe7cc9e770b50eb/exceptiongroup-1.3.0.tar.gz", hash = "sha256:b241f5885f560bc56a59ee63ca4c6a8bfa46ae4ad651af316d4e81817bb9fd88", size = 29749, upload-time = "2025-05-10T17:42:51.123Z" } @@ -548,56 +444,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/36/f4/c6e662dade71f56cd2f3735141b265c3c79293c109549c1e6933b0651ffc/exceptiongroup-1.3.0-py3-none-any.whl", hash = "sha256:4d111e6e0c13d0644cad6ddaa7ed0261a0b36971f6d23e7ec9b4b9097da78a10", size = 16674, upload-time = "2025-05-10T17:42:49.33Z" }, ] -[[package]] -name = "execnet" -version = "2.0.2" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/e4/c8/d382dc7a1e68a165f4a4ab612a08b20d8534a7d20cc590630b734ca0c54b/execnet-2.0.2.tar.gz", hash = "sha256:cc59bc4423742fd71ad227122eb0dd44db51efb3dc4095b45ac9a08c770096af", size = 161098, upload-time = "2023-07-09T17:14:03.468Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/e8/9c/a079946da30fac4924d92dbc617e5367d454954494cf1e71567bcc4e00ee/execnet-2.0.2-py3-none-any.whl", hash = "sha256:88256416ae766bc9e8895c76a87928c0012183da3cc4fc18016e6f050e025f41", size = 37097, upload-time = "2023-07-09T17:14:01.888Z" }, -] - [[package]] name = "execnet" version = "2.1.2" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" } wheels = [ { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" }, ] -[[package]] -name = "h11" -version = "0.14.0" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -dependencies = [ - { name = "typing-extensions", version = "4.7.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/f5/38/3af3d3633a34a3316095b39c8e8fb4853a28a536e55d347bd8d8e9a14b03/h11-0.14.0.tar.gz", hash = "sha256:8f19fbbe99e72420ff35c00b27a34cb9937e902a8b810e2c88300c6f0a3b699d", size = 100418, upload-time = "2022-09-25T15:40:01.519Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/95/04/ff642e65ad6b90db43e668d70ffb6736436c7ce41fcc549f4e9472234127/h11-0.14.0-py3-none-any.whl", hash = "sha256:e3fe4ac4b851c468cc8363d500db52c2ead036020723024a109d37346efaa761", size = 58259, upload-time = "2022-09-25T15:39:59.68Z" }, -] - [[package]] name = "h11" version = "0.16.0" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } wheels = [ { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, @@ -608,8 +467,7 @@ name = "h2" version = "4.1.0" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", - "python_full_version < '3.8'", + "python_full_version < '3.9'", ] dependencies = [ { name = "hpack", version = "4.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, @@ -642,8 +500,7 @@ name = "hpack" version = "4.0.0" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", - "python_full_version < '3.8'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/3e/9b/fda93fb4d957db19b0f6b370e79d586b3e8528b20252c729c476a2c02954/hpack-4.0.0.tar.gz", hash = "sha256:fc41de0c63e687ebffde81187a948221294896f6bdc0ae2312708df339430095", size = 49117, upload-time = "2020-08-30T10:35:57.868Z" } wheels = [ @@ -663,75 +520,29 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/07/c6/80c95b1b2b94682a72cbdbfb85b81ae2daffa4291fbfa1b1464502ede10d/hpack-4.1.0-py3-none-any.whl", hash = "sha256:157ac792668d995c657d93111f46b4535ed114f0c9c8d672271bbec7eae1b496", size = 34357, upload-time = "2025-01-22T21:44:56.92Z" }, ] -[[package]] -name = "httpcore" -version = "0.17.3" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -dependencies = [ - { name = "anyio", version = "3.7.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "certifi", marker = "python_full_version < '3.8'" }, - { name = "h11", version = "0.14.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "sniffio", marker = "python_full_version < '3.8'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/63/ad/c98ecdbfe04417e71e143bf2f2fb29128e4787d78d1cedba21bd250c7e7a/httpcore-0.17.3.tar.gz", hash = "sha256:a6f30213335e34c1ade7be6ec7c47f19f50c56db36abef1a9dfa3815b1cb3888", size = 62676, upload-time = "2023-07-05T12:09:31.29Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/94/2c/2bde7ff8dd2064395555220cbf7cba79991172bf5315a07eb3ac7688d9f1/httpcore-0.17.3-py3-none-any.whl", hash = "sha256:c2789b767ddddfa2a5782e3199b2b7f6894540b17b16ec26b2c4d8e103510b87", size = 74513, upload-time = "2023-07-05T12:09:29.425Z" }, -] - [[package]] name = "httpcore" version = "1.0.9" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] dependencies = [ - { name = "certifi", marker = "python_full_version >= '3.8'" }, - { name = "h11", version = "0.16.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, + { name = "certifi" }, + { name = "h11" }, ] sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } wheels = [ { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, ] -[[package]] -name = "httpx" -version = "0.24.1" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -dependencies = [ - { name = "certifi", marker = "python_full_version < '3.8'" }, - { name = "httpcore", version = "0.17.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "idna", version = "3.10", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "sniffio", marker = "python_full_version < '3.8'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/f8/2a/114d454cb77657dbf6a293e69390b96318930ace9cd96b51b99682493276/httpx-0.24.1.tar.gz", hash = "sha256:5853a43053df830c20f8110c5e69fe44d035d850b2dfe795e196f00fdb774bdd", size = 81858, upload-time = "2023-05-19T00:50:56.678Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ec/91/e41f64f03d2a13aee7e8c819d82ee3aa7cdc484d18c0ae859742597d5aa0/httpx-0.24.1-py3-none-any.whl", hash = "sha256:06781eb9ac53cde990577af654bd990a4949de37a28bdb4a230d434f3a30b9bd", size = 75377, upload-time = "2023-05-19T00:50:54.91Z" }, -] - [[package]] name = "httpx" version = "0.28.1" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] dependencies = [ - { name = "anyio", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "anyio", version = "4.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "anyio", version = "4.11.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, - { name = "certifi", marker = "python_full_version >= '3.8'" }, - { name = "httpcore", version = "1.0.9", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, - { name = "idna", version = "3.11", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, ] sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } wheels = [ @@ -743,8 +554,7 @@ name = "hyperframe" version = "6.0.1" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", - "python_full_version < '3.8'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/5a/2a/4747bff0a17f7281abe73e955d60d80aae537a5d203f417fa1c2e7578ebb/hyperframe-6.0.1.tar.gz", hash = "sha256:ae510046231dc8e9ecb1a6586f63d2347bf4c8905914aa84ba585ae85f28a914", size = 25008, upload-time = "2021-04-17T12:11:22.757Z" } wheels = [ @@ -764,27 +574,10 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/48/30/47d0bf6072f7252e6521f3447ccfa40b421b6824517f82854703d0f5a98b/hyperframe-6.1.0-py3-none-any.whl", hash = "sha256:b03380493a519fce58ea5af42e4a42317bf9bd425596f7a0835ffce80f1a42e5", size = 13007, upload-time = "2025-01-22T21:41:47.295Z" }, ] -[[package]] -name = "idna" -version = "3.10" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/f1/70/7703c29685631f5a7590aa73f1f1d3fa9a380e654b86af429e0934a32f7d/idna-3.10.tar.gz", hash = "sha256:12f65c9b470abda6dc35cf8e63cc574b1c52b11df2c86030af0ac09b01b13ea9", size = 190490, upload-time = "2024-09-15T18:07:39.745Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl", hash = "sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3", size = 70442, upload-time = "2024-09-15T18:07:37.964Z" }, -] - [[package]] name = "idna" version = "3.11" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/0703ccc57f3a7233505399edb88de3cbd678da106337b9fcde432b65ed60/idna-3.11.tar.gz", hash = "sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902", size = 194582, upload-time = "2025-10-12T14:55:20.501Z" } wheels = [ { url = "https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl", hash = "sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea", size = 71008, upload-time = "2025-10-12T14:55:18.883Z" }, @@ -795,9 +588,7 @@ name = "importlib-metadata" version = "4.13.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", version = "4.7.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "zipp", version = "3.15.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "zipp", version = "3.20.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "zipp", version = "3.20.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "zipp", version = "3.23.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/55/12/ab288357b884ebc807e3f4eff63ce5ba6b941ba61499071bf19f1bbc7f7f/importlib_metadata-4.13.0.tar.gz", hash = "sha256:dd0173e8f150d6815e098fd354f6414b0f079af4644ddfe90c71e2fc6174346d", size = 50445, upload-time = "2022-10-01T17:09:15.698Z" } @@ -805,25 +596,13 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d0/98/c277899f5aa21f6e6946e1c83f2af650cbfee982763ffb91db07ff7d3a13/importlib_metadata-4.13.0-py3-none-any.whl", hash = "sha256:8a8a81bcf996e74fee46f0d16bd3eaa382a7eb20fd82445c3ad11f4090334116", size = 23010, upload-time = "2022-10-01T17:09:13.903Z" }, ] -[[package]] -name = "iniconfig" -version = "2.0.0" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/d7/4b/cbd8e699e64a6f16ca3a8220661b5f83792b3017d0f79807cb8708d33913/iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3", size = 4646, upload-time = "2023-01-07T11:08:11.254Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374", size = 5892, upload-time = "2023-01-07T11:08:09.864Z" }, -] - [[package]] name = "iniconfig" version = "2.1.0" source = { registry = "https://pypi.org/simple" } resolution-markers = [ "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/f2/97/ebf4da567aa6827c909642694d71c9fcf53e5b504f2d96afea02718862f3/iniconfig-2.1.0.tar.gz", hash = "sha256:3abbd2e30b36733fee78f9c7f7308f2d0050e88f0087fd25c2645f63c773e1c7", size = 4793, upload-time = "2025-03-19T20:09:59.721Z" } wheels = [ @@ -851,76 +630,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/5c/03/b7e605db4a57c0f6fba744b11ef3ddf4ddebcada35022927a2b5fc623fdf/mock-4.0.3-py3-none-any.whl", hash = "sha256:122fcb64ee37cfad5b3f48d7a7d51875d7031aaf3d8be7c42e2bee25044eee62", size = 28536, upload-time = "2020-12-10T07:33:11.564Z" }, ] -[[package]] -name = "msgpack" -version = "1.0.5" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/dc/a1/eba11a0d4b764bc62966a565b470f8c6f38242723ba3057e9b5098678c30/msgpack-1.0.5.tar.gz", hash = "sha256:c075544284eadc5cddc70f4757331d99dcbc16b2bbd4849d15f8aae4cf36d31c", size = 127834, upload-time = "2023-03-08T17:50:48.564Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/9f/4a/36d936e54cf71e23ad276564465f6a54fb129e3d61520b76e13e0bb29167/msgpack-1.0.5-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:525228efd79bb831cf6830a732e2e80bc1b05436b086d4264814b4b2955b2fa9", size = 129738, upload-time = "2023-03-08T17:49:18.464Z" }, - { url = "https://files.pythonhosted.org/packages/f2/da/770118f8d48e11cc9a2c7cb60d7d3c8016266526bd42c6ff5bd21013d099/msgpack-1.0.5-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:4f8d8b3bf1ff2672567d6b5c725a1b347fe838b912772aa8ae2bf70338d5a198", size = 74671, upload-time = "2023-03-08T17:49:20.311Z" }, - { url = "https://files.pythonhosted.org/packages/73/99/f338ce8b69e934c04e5d9187f85de1ae395882cd56e7deb48e78a1749af8/msgpack-1.0.5-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:cdc793c50be3f01106245a61b739328f7dccc2c648b501e237f0699fe1395b81", size = 70230, upload-time = "2023-03-08T17:49:21.958Z" }, - { url = "https://files.pythonhosted.org/packages/3e/80/bc7fdb75a35bf32c7c529c247dcadfd0502aac2309e207a89b0be6fe42ea/msgpack-1.0.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5cb47c21a8a65b165ce29f2bec852790cbc04936f502966768e4aae9fa763cb7", size = 309410, upload-time = "2023-03-08T17:49:23.337Z" }, - { url = "https://files.pythonhosted.org/packages/59/67/f992ada3b42889f1b984e5651d63ea21ca3a92049cff6d75fe0a4a63e422/msgpack-1.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e42b9594cc3bf4d838d67d6ed62b9e59e201862a25e9a157019e171fbe672dd3", size = 316846, upload-time = "2023-03-08T17:49:24.786Z" }, - { url = "https://files.pythonhosted.org/packages/10/fe/9e004c4deb457f1ef1ad88c1188da5691ff1855e0d03a5ac3635ae1f6530/msgpack-1.0.5-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:55b56a24893105dc52c1253649b60f475f36b3aa0fc66115bffafb624d7cb30b", size = 311396, upload-time = "2023-03-08T17:49:26.075Z" }, - { url = "https://files.pythonhosted.org/packages/95/c9/560c3203c4327881c9f2de26c42dacdd9567bfe7fa43458e2a680c4bdcaf/msgpack-1.0.5-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:1967f6129fc50a43bfe0951c35acbb729be89a55d849fab7686004da85103f1c", size = 311165, upload-time = "2023-03-08T17:49:27.494Z" }, - { url = "https://files.pythonhosted.org/packages/10/ca/50c3a5e92d459a942169747315afd8c226d05427eccff903ddf33135c574/msgpack-1.0.5-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:20a97bf595a232c3ee6d57ddaadd5453d174a52594bf9c21d10407e2a2d9b3bd", size = 348664, upload-time = "2023-03-08T17:49:28.736Z" }, - { url = "https://files.pythonhosted.org/packages/29/56/1fb6b96aab759ab3bc05b03ba6d936b350db72aac203cde56ea6bd001237/msgpack-1.0.5-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:d25dd59bbbbb996eacf7be6b4ad082ed7eacc4e8f3d2df1ba43822da9bfa122a", size = 316731, upload-time = "2023-03-08T17:49:30.041Z" }, - { url = "https://files.pythonhosted.org/packages/7b/e9/b47f9e93fc381885624c40cbbbd0480b18ae11ca588162fe724d43495372/msgpack-1.0.5-cp310-cp310-win32.whl", hash = "sha256:382b2c77589331f2cb80b67cc058c00f225e19827dbc818d700f61513ab47bea", size = 57134, upload-time = "2023-03-08T17:49:31.365Z" }, - { url = "https://files.pythonhosted.org/packages/3c/e5/3d436bed11849ba05d777ed3fd1a0440170bad460335ea541dd6946047ed/msgpack-1.0.5-cp310-cp310-win_amd64.whl", hash = "sha256:4867aa2df9e2a5fa5f76d7d5565d25ec76e84c106b55509e78c1ede0f152659a", size = 61631, upload-time = "2023-03-08T17:49:32.482Z" }, - { url = "https://files.pythonhosted.org/packages/27/ad/4edfe383ec3185611441179ffee8cbc8155d7575fbad73f6d31015e35451/msgpack-1.0.5-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:9f5ae84c5c8a857ec44dc180a8b0cc08238e021f57abdf51a8182e915e6299f0", size = 127502, upload-time = "2023-03-08T17:49:33.974Z" }, - { url = "https://files.pythonhosted.org/packages/c1/57/01f2d8805160f559ec21d095fc7576a26fbaed2475af24ce4a135c380c14/msgpack-1.0.5-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:9e6ca5d5699bcd89ae605c150aee83b5321f2115695e741b99618f4856c50898", size = 73747, upload-time = "2023-03-08T17:49:35.158Z" }, - { url = "https://files.pythonhosted.org/packages/6c/fe/8a7747ca57074307a2e8f1de58441952a9dbdf9e8a8e5873d53a5ce0835c/msgpack-1.0.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:5494ea30d517a3576749cad32fa27f7585c65f5f38309c88c6d137877fa28a5a", size = 69041, upload-time = "2023-03-08T17:49:36.44Z" }, - { url = "https://files.pythonhosted.org/packages/33/0a/aa7b53ae17cf1dc1c352d705ab3162fc572c55048cc3177c1a88009c47fd/msgpack-1.0.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1ab2f3331cb1b54165976a9d976cb251a83183631c88076613c6c780f0d6e45a", size = 316114, upload-time = "2023-03-08T17:49:38.252Z" }, - { url = "https://files.pythonhosted.org/packages/6b/6d/de239d77d347f1990c41b4800075a15e06f748186dd120166270dd071734/msgpack-1.0.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:28592e20bbb1620848256ebc105fc420436af59515793ed27d5c77a217477705", size = 325080, upload-time = "2023-03-08T17:49:39.528Z" }, - { url = "https://files.pythonhosted.org/packages/7e/1c/9d0fd241a4e88e1cd2f5babea4a27ac25b1b86dbbc05fa10741e82079a93/msgpack-1.0.5-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:fe5c63197c55bce6385d9aee16c4d0641684628f63ace85f73571e65ad1c1e8d", size = 319393, upload-time = "2023-03-08T17:49:40.755Z" }, - { url = "https://files.pythonhosted.org/packages/b8/bc/1d5fe4732dc78ff86aaf677596da08f0ae736e60ca8ab49c1f1c7366cb1a/msgpack-1.0.5-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:ed40e926fa2f297e8a653c954b732f125ef97bdd4c889f243182299de27e2aa9", size = 316118, upload-time = "2023-03-08T17:49:41.964Z" }, - { url = "https://files.pythonhosted.org/packages/2c/e9/c79ecc36cfa34d850a01773565e0fccafd69efff07172028c3a5f758b83f/msgpack-1.0.5-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:b2de4c1c0538dcb7010902a2b97f4e00fc4ddf2c8cda9749af0e594d3b7fa3d7", size = 354984, upload-time = "2023-03-08T17:49:43.251Z" }, - { url = "https://files.pythonhosted.org/packages/45/85/6b55b0cabad846d3e730226a897f878f8f63ee505668bb6c55a697b0bfb0/msgpack-1.0.5-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:bf22a83f973b50f9d38e55c6aade04c41ddda19b00c4ebc558930d78eecc64ed", size = 323580, upload-time = "2023-03-08T17:49:44.71Z" }, - { url = "https://files.pythonhosted.org/packages/0e/69/3d10e741dd2bbb806af5cdc76551735baab5f5f9773701eb05502c913a6e/msgpack-1.0.5-cp311-cp311-win32.whl", hash = "sha256:c396e2cc213d12ce017b686e0f53497f94f8ba2b24799c25d913d46c08ec422c", size = 56419, upload-time = "2023-03-08T17:49:46.603Z" }, - { url = "https://files.pythonhosted.org/packages/6b/79/0dec8f035160464ca88b221cc79691a71cf88dc25207c17f1d918b2c7bb0/msgpack-1.0.5-cp311-cp311-win_amd64.whl", hash = "sha256:6c4c68d87497f66f96d50142a2b73b97972130d93677ce930718f68828b382e2", size = 60781, upload-time = "2023-03-08T17:49:47.912Z" }, - { url = "https://files.pythonhosted.org/packages/c5/c1/1b591574ba71481fbf38359a8fca5108e4ad130a6dbb9b2acb3e9277d0fe/msgpack-1.0.5-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:137850656634abddfb88236008339fdaba3178f4751b28f270d2ebe77a563b6c", size = 72520, upload-time = "2023-03-08T17:50:02.199Z" }, - { url = "https://files.pythonhosted.org/packages/62/57/170af6c6fccd2d950ea01e1faa58cae9643226fa8705baded11eca3aa8b5/msgpack-1.0.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0c05a4a96585525916b109bb85f8cb6511db1c6f5b9d9cbcbc940dc6b4be944b", size = 289288, upload-time = "2023-03-08T17:50:04.213Z" }, - { url = "https://files.pythonhosted.org/packages/2b/c4/f2c8695ae69d1425eddc5e2f849c525b562dc8409bc2979e525f3dd4fecd/msgpack-1.0.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:56a62ec00b636583e5cb6ad313bbed36bb7ead5fa3a3e38938503142c72cba4f", size = 299695, upload-time = "2023-03-08T17:50:05.622Z" }, - { url = "https://files.pythonhosted.org/packages/62/5c/9c7fed4ca0235a2d7b8d15b4047c328976b97d2b227719e54cad1e47c244/msgpack-1.0.5-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ef8108f8dedf204bb7b42994abf93882da1159728a2d4c5e82012edd92c9da9f", size = 293149, upload-time = "2023-03-08T17:50:06.915Z" }, - { url = "https://files.pythonhosted.org/packages/ef/13/c110d89d5079169354394dc226e6f84d818722939bc1fe3f9c25f982e903/msgpack-1.0.5-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:1835c84d65f46900920b3708f5ba829fb19b1096c1800ad60bae8418652a951d", size = 292899, upload-time = "2023-03-08T17:50:08.215Z" }, - { url = "https://files.pythonhosted.org/packages/72/ac/2eda5af7cd1450c52d031e48c76b280eac5bb2e588678876612f95be34ab/msgpack-1.0.5-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:e57916ef1bd0fee4f21c4600e9d1da352d8816b52a599c46460e93a6e9f17086", size = 334408, upload-time = "2023-03-08T17:50:10Z" }, - { url = "https://files.pythonhosted.org/packages/e8/60/78906f564804aae23eb1102eca8b8830f1e08a649c179774c05fa7dc0aad/msgpack-1.0.5-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:17358523b85973e5f242ad74aa4712b7ee560715562554aa2134d96e7aa4cbbf", size = 302791, upload-time = "2023-03-08T17:50:11.414Z" }, - { url = "https://files.pythonhosted.org/packages/ce/b8/89cb1809b076a4651169851aa1f98128b75cbfe14034b914c9040b13c4cf/msgpack-1.0.5-cp37-cp37m-win32.whl", hash = "sha256:cb5aaa8c17760909ec6cb15e744c3ebc2ca8918e727216e79607b7bbce9c8f77", size = 57095, upload-time = "2023-03-08T17:50:12.741Z" }, - { url = "https://files.pythonhosted.org/packages/e8/1f/be19c9c9cfdcc2ae8ee8c65dbe5f281cc1f3331f9b9523735f39b090b448/msgpack-1.0.5-cp37-cp37m-win_amd64.whl", hash = "sha256:ab31e908d8424d55601ad7075e471b7d0140d4d3dd3272daf39c5c19d936bd82", size = 62112, upload-time = "2023-03-08T17:50:13.947Z" }, - { url = "https://files.pythonhosted.org/packages/1a/f7/df5814697c25bdebb14ea97d27ddca04f5d4c6e249f096d086fea521c139/msgpack-1.0.5-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:b72d0698f86e8d9ddf9442bdedec15b71df3598199ba33322d9711a19f08145c", size = 126923, upload-time = "2023-03-08T17:50:15.214Z" }, - { url = "https://files.pythonhosted.org/packages/33/52/099f0dde1283bac7bf267ab941dfa3b7c89ee701e4252973f8d3c10e68d6/msgpack-1.0.5-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:379026812e49258016dd84ad79ac8446922234d498058ae1d415f04b522d5b2d", size = 73246, upload-time = "2023-03-08T17:50:16.487Z" }, - { url = "https://files.pythonhosted.org/packages/f1/1f/cc3e8274934c8323f6106dae22cba8bad413166f4efb3819573de58c215c/msgpack-1.0.5-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:332360ff25469c346a1c5e47cbe2a725517919892eda5cfaffe6046656f0b7bb", size = 68947, upload-time = "2023-03-08T17:50:17.767Z" }, - { url = "https://files.pythonhosted.org/packages/19/0c/2c3b443df88f5d400f2e19a3d867564d004b26e137f18c2f2663913987bc/msgpack-1.0.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:476a8fe8fae289fdf273d6d2a6cb6e35b5a58541693e8f9f019bfe990a51e4ba", size = 313568, upload-time = "2023-03-08T17:50:19.016Z" }, - { url = "https://files.pythonhosted.org/packages/0a/04/bc319ba061f6dc9077745988be288705b3f9f18c5a209772a3e8fcd419fd/msgpack-1.0.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a9985b214f33311df47e274eb788a5893a761d025e2b92c723ba4c63936b69b1", size = 322443, upload-time = "2023-03-08T17:50:20.341Z" }, - { url = "https://files.pythonhosted.org/packages/2f/21/e488871f8e498efe14821b0c870eb95af52cfafb9b8dd41d83fad85b383b/msgpack-1.0.5-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:48296af57cdb1d885843afd73c4656be5c76c0c6328db3440c9601a98f303d87", size = 315490, upload-time = "2023-03-08T17:50:22.301Z" }, - { url = "https://files.pythonhosted.org/packages/28/8f/c58c53c884217cc572c19349c7e1129b5a6eae36df0a017aae3a8f3d7aa8/msgpack-1.0.5-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:addab7e2e1fcc04bd08e4eb631c2a90960c340e40dfc4a5e24d2ff0d5a3b3edb", size = 324288, upload-time = "2023-03-08T17:50:23.739Z" }, - { url = "https://files.pythonhosted.org/packages/0d/90/44edef4a8c6f035b054c4b017c5adcb22a35ec377e17e50dd5dced279a6b/msgpack-1.0.5-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:916723458c25dfb77ff07f4c66aed34e47503b2eb3188b3adbec8d8aa6e00f48", size = 361405, upload-time = "2023-03-08T17:50:24.992Z" }, - { url = "https://files.pythonhosted.org/packages/56/50/bfcc0fad07067b6f1b09d940272ec749d5fe82570d938c2348c3ad0babf7/msgpack-1.0.5-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:821c7e677cc6acf0fd3f7ac664c98803827ae6de594a9f99563e48c5a2f27eb0", size = 329585, upload-time = "2023-03-08T17:50:26.387Z" }, - { url = "https://files.pythonhosted.org/packages/80/f0/c1fadb4e4a38fda19e35b1b6f887d72cc9c57778af43b53f64a8cd62e922/msgpack-1.0.5-cp38-cp38-win32.whl", hash = "sha256:1c0f7c47f0087ffda62961d425e4407961a7ffd2aa004c81b9c07d9269512f6e", size = 57668, upload-time = "2023-03-08T17:50:28.037Z" }, - { url = "https://files.pythonhosted.org/packages/da/46/855bdcbf004fd87b6a4451e8dcd61329439dcd9039887f71ca5085769216/msgpack-1.0.5-cp38-cp38-win_amd64.whl", hash = "sha256:bae7de2026cbfe3782c8b78b0db9cbfc5455e079f1937cb0ab8d133496ac55e1", size = 62509, upload-time = "2023-03-08T17:50:29.85Z" }, - { url = "https://files.pythonhosted.org/packages/12/6e/0cfd1dc07f61a6ac606587a393f489c3ca463469d285a73c8e5e2f61b021/msgpack-1.0.5-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:20c784e66b613c7f16f632e7b5e8a1651aa5702463d61394671ba07b2fc9e025", size = 130498, upload-time = "2023-03-08T17:50:31.551Z" }, - { url = "https://files.pythonhosted.org/packages/4b/3d/cc5eb6d69e0ecde80a78cc42f48579971ec333e509d56a4a6de1a2c40ba2/msgpack-1.0.5-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:266fa4202c0eb94d26822d9bfd7af25d1e2c088927fe8de9033d929dd5ba24c5", size = 75178, upload-time = "2023-03-08T17:50:32.78Z" }, - { url = "https://files.pythonhosted.org/packages/bf/68/032e62ad44f92ba6a4ae7c45054843cdec7f0c405ecdfd166f25123b0c47/msgpack-1.0.5-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:18334484eafc2b1aa47a6d42427da7fa8f2ab3d60b674120bce7a895a0a85bdd", size = 70460, upload-time = "2023-03-08T17:50:34.57Z" }, - { url = "https://files.pythonhosted.org/packages/49/57/a28120d82f8e77622a1e1efc652389c71145f6b89b47b39814a7c6038373/msgpack-1.0.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:57e1f3528bd95cc44684beda696f74d3aaa8a5e58c816214b9046512240ef437", size = 313499, upload-time = "2023-03-08T17:50:36.329Z" }, - { url = "https://files.pythonhosted.org/packages/ab/ff/ca74e519c47139b6c08fb21db5ead2bd2eed6cb1225f9be69390cdb48182/msgpack-1.0.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:586d0d636f9a628ddc6a17bfd45aa5b5efaf1606d2b60fa5d87b8986326e933f", size = 322301, upload-time = "2023-03-08T17:50:38.097Z" }, - { url = "https://files.pythonhosted.org/packages/43/87/6507d56f62b958d822ae4ffe1c4507ed7d3cf37ad61114665816adcf4adc/msgpack-1.0.5-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a740fa0e4087a734455f0fc3abf5e746004c9da72fbd541e9b113013c8dc3282", size = 316630, upload-time = "2023-03-08T17:50:39.397Z" }, - { url = "https://files.pythonhosted.org/packages/67/f8/e3ab674f4a945308362e9342297fe6b35a89dd0f648aa325aabffa5dc210/msgpack-1.0.5-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:3055b0455e45810820db1f29d900bf39466df96ddca11dfa6d074fa47054376d", size = 316251, upload-time = "2023-03-08T17:50:41.153Z" }, - { url = "https://files.pythonhosted.org/packages/e9/f1/45b73a9e97f702bcb5f51569b93990e456bc969363e55122374c22ed7d24/msgpack-1.0.5-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:a61215eac016f391129a013c9e46f3ab308db5f5ec9f25811e811f96962599a8", size = 352781, upload-time = "2023-03-08T17:50:42.435Z" }, - { url = "https://files.pythonhosted.org/packages/17/10/be97811782473d709d07b65a3955a5a76d47686aff3d62bb41d48aea7c92/msgpack-1.0.5-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:362d9655cd369b08fda06b6657a303eb7172d5279997abe094512e919cf74b11", size = 321996, upload-time = "2023-03-08T17:50:43.726Z" }, - { url = "https://files.pythonhosted.org/packages/18/3f/3860151fbdf50e369bbe4ffd307a588417669c725025e383f3ce5893690f/msgpack-1.0.5-cp39-cp39-win32.whl", hash = "sha256:ac9dd47af78cae935901a9a500104e2dea2e253207c924cc95de149606dc43cc", size = 57827, upload-time = "2023-03-08T17:50:45.517Z" }, - { url = "https://files.pythonhosted.org/packages/6c/fa/3ca00fb1e53bcacf8c186fa6aff2d2086862b12e289bcf38227d9d40bd86/msgpack-1.0.5-cp39-cp39-win_amd64.whl", hash = "sha256:06f5174b5f8ed0ed919da0e62cbd4ffde676a374aba4020034da05fab67b9164", size = 62775, upload-time = "2023-03-08T17:50:47.305Z" }, -] - [[package]] name = "msgpack" version = "1.1.1" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/45/b1/ea4f68038a18c77c9467400d166d74c4ffa536f34761f7983a104357e614/msgpack-1.1.1.tar.gz", hash = "sha256:77b79ce34a2bdab2594f490c8e80dd62a02d650b91a75159a63ec413b8d104cd", size = 173555, upload-time = "2025-06-13T06:52:51.324Z" } wheels = [ @@ -1057,53 +772,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/14/a7/b1992b4fb3da3b413f5fb78a63bad42f256c3be2352eb69273c3789c2c96/msgpack-1.1.2-cp39-cp39-win_amd64.whl", hash = "sha256:67016ae8c8965124fdede9d3769528ad8284f14d635337ffa6a713a580f6c030", size = 71540, upload-time = "2025-10-08T09:15:55.573Z" }, ] -[[package]] -name = "packaging" -version = "24.0" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/ee/b5/b43a27ac7472e1818c4bafd44430e69605baefe1f34440593e0332ec8b4d/packaging-24.0.tar.gz", hash = "sha256:eb82c5e3e56209074766e6885bb04b8c38a0c015d0a30036ebe7ece34c9989e9", size = 147882, upload-time = "2024-03-10T09:39:28.33Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl", hash = "sha256:2ddfb553fdf02fb784c234c7ba6ccc288296ceabec964ad2eae3777778130bc5", size = 53488, upload-time = "2024-03-10T09:39:25.947Z" }, -] - [[package]] name = "packaging" version = "25.0" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] sdist = { url = "https://files.pythonhosted.org/packages/a1/d4/1fc4078c65507b51b96ca8f8c3ba19e6a61c8253c72794544580a7b6c24d/packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f", size = 165727, upload-time = "2025-04-19T11:48:59.673Z" } wheels = [ { url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" }, ] -[[package]] -name = "pluggy" -version = "1.2.0" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -dependencies = [ - { name = "importlib-metadata", marker = "python_full_version < '3.8'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/8a/42/8f2833655a29c4e9cb52ee8a2be04ceac61bcff4a680fb338cbd3d1e322d/pluggy-1.2.0.tar.gz", hash = "sha256:d12f0c4b579b15f5e054301bb226ee85eeeba08ffec228092f8defbaa3a4c4b3", size = 61613, upload-time = "2023-06-21T09:12:28.745Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/51/32/4a79112b8b87b21450b066e102d6608907f4c885ed7b04c3fdb085d4d6ae/pluggy-1.2.0-py3-none-any.whl", hash = "sha256:c2fd55a7d7a3863cba1a013e4e2414658b1d07b6bc57b3919e0c63c9abb99849", size = 17695, upload-time = "2023-06-21T09:12:27.397Z" }, -] - [[package]] name = "pluggy" version = "1.5.0" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/96/2d/02d4312c973c6050a18b314a5ad0b3210edb65a906f868e31c111dede4a6/pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1", size = 67955, upload-time = "2024-04-20T21:34:42.531Z" } wheels = [ @@ -1178,32 +861,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2b/62/2392b7879f4d2c1bfa20815720b89d464687877851716936b9609959c201/pycryptodome-3.23.0-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:ce64e84a962b63a47a592690bdc16a7eaf709d2c2697ababf24a0def566899a6", size = 1802461, upload-time = "2025-05-17T17:21:41.722Z" }, ] -[[package]] -name = "pyee" -version = "9.1.1" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -dependencies = [ - { name = "typing-extensions", version = "4.7.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/eb/2c/ebe4fd8213b3d720b193a62f07169607e945dd02a08edc45b28ca52fbe07/pyee-9.1.1.tar.gz", hash = "sha256:a1ebcd38d92e7d780635ab3442c0f6ce49840d9bfe0b0549921a6188860af0db", size = 22634, upload-time = "2023-06-09T06:13:29.141Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/f2/53/39b67ce3841a5bb2d444f64ed969fb79ebd5bfed6867c3f88f3916407270/pyee-9.1.1-py2.py3-none-any.whl", hash = "sha256:f4a9853503d2f5a69d4350b54ba70841ebc535c53ebfaaa40c0fb47e63e78b3e", size = 15073, upload-time = "2023-06-09T06:13:27.255Z" }, -] - [[package]] name = "pyee" version = "13.0.0" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] dependencies = [ - { name = "typing-extensions", version = "4.13.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "typing-extensions", version = "4.13.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "typing-extensions", version = "4.15.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/95/03/1fd98d5841cd7964a27d729ccf2199602fe05eb7a405c1462eb7277945ed/pyee-13.0.0.tar.gz", hash = "sha256:b391e3c5a434d1f5118a25615001dbc8f669cf410ab67d04c4d4e07c55481c37", size = 31250, upload-time = "2025-03-17T18:53:15.955Z" } @@ -1218,50 +881,24 @@ source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "colorama", marker = "sys_platform == 'win32'" }, { name = "exceptiongroup", marker = "python_full_version < '3.11'" }, - { name = "importlib-metadata", marker = "python_full_version < '3.8'" }, - { name = "iniconfig", version = "2.0.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "iniconfig", version = "2.1.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8' and python_full_version < '3.10'" }, + { name = "iniconfig", version = "2.1.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" }, { name = "iniconfig", version = "2.3.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, - { name = "packaging", version = "24.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, - { name = "pluggy", version = "1.2.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "pluggy", version = "1.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "packaging" }, + { name = "pluggy", version = "1.5.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "pluggy", version = "1.6.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.9'" }, - { name = "tomli", version = "2.0.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "tomli", version = "2.3.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8' and python_full_version < '3.11'" }, + { name = "tomli", marker = "python_full_version < '3.11'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/80/1f/9d8e98e4133ffb16c90f3b405c43e38d3abb715bb5d7a63a5a684f7e46a3/pytest-7.4.4.tar.gz", hash = "sha256:2cf0005922c6ace4a3e2ec8b4080eb0d9753fdc93107415332f50ce9e7994280", size = 1357116, upload-time = "2023-12-31T12:00:18.035Z" } wheels = [ { url = "https://files.pythonhosted.org/packages/51/ff/f6e8b8f39e08547faece4bd80f89d5a8de68a38b2d179cc1c4490ffa3286/pytest-7.4.4-py3-none-any.whl", hash = "sha256:b090cdf5ed60bf4c45261be03239c2c1c22df034fbffe691abe93cd80cea01d8", size = 325287, upload-time = "2023-12-31T12:00:13.963Z" }, ] -[[package]] -name = "pytest-asyncio" -version = "0.21.2" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -dependencies = [ - { name = "pytest", marker = "python_full_version < '3.8'" }, - { name = "typing-extensions", version = "4.7.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/ae/53/57663d99acaac2fcdafdc697e52a9b1b7d6fcf36616281ff9768a44e7ff3/pytest_asyncio-0.21.2.tar.gz", hash = "sha256:d67738fc232b94b326b9d060750beb16e0074210b98dd8b58a5239fa2a154f45", size = 30656, upload-time = "2024-04-29T13:23:24.738Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/9c/ce/1e4b53c213dce25d6e8b163697fbce2d43799d76fa08eea6ad270451c370/pytest_asyncio-0.21.2-py3-none-any.whl", hash = "sha256:ab664c88bb7998f711d8039cacd4884da6430886ae8bbd4eded552ed2004f16b", size = 13368, upload-time = "2024-04-29T13:23:23.126Z" }, -] - [[package]] name = "pytest-asyncio" version = "0.23.8" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] dependencies = [ - { name = "pytest", marker = "python_full_version >= '3.8'" }, + { name = "pytest" }, ] sdist = { url = "https://files.pythonhosted.org/packages/de/b4/0b378b7bf26a8ae161c3890c0b48a91a04106c5713ce81b4b080ea2f4f18/pytest_asyncio-0.23.8.tar.gz", hash = "sha256:759b10b33a6dc61cce40a8bd5205e302978bbbcc00e279a8b61d9a6a3c82e4d3", size = 46920, upload-time = "2024-07-17T17:39:34.617Z" } wheels = [ @@ -1273,8 +910,7 @@ name = "pytest-cov" version = "2.12.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "coverage", version = "7.2.7", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "coverage", version = "7.6.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.8.*'" }, + { name = "coverage", version = "7.6.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.9'" }, { name = "coverage", version = "7.10.7", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version == '3.9.*'" }, { name = "coverage", version = "7.12.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.10'" }, { name = "pytest" }, @@ -1315,8 +951,7 @@ name = "pytest-xdist" version = "1.34.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "execnet", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, - { name = "execnet", version = "2.1.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, + { name = "execnet" }, { name = "pytest" }, { name = "pytest-forked" }, { name = "six" }, @@ -1326,32 +961,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/6a/fc/30821e7799bddd56989523ee003cde488c6e6053dfd29ba07db2ba934a04/pytest_xdist-1.34.0-py2.py3-none-any.whl", hash = "sha256:ba5d10729372d65df3ac150872f9df5d2ed004a3b0d499cc0164aafedd8c7b66", size = 36841, upload-time = "2020-07-27T23:05:23.851Z" }, ] -[[package]] -name = "respx" -version = "0.20.2" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -dependencies = [ - { name = "httpx", version = "0.24.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.8'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/e6/0b/e0df26ea5c7145d95f1ab8ecb20f0778dd8af718e56747977dca9d28362a/respx-0.20.2.tar.gz", hash = "sha256:07cf4108b1c88b82010f67d3c831dae33a375c7b436e54d87737c7f9f99be643", size = 26080, upload-time = "2023-07-20T23:01:23.618Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/47/47/8c5a8b02c2144770fe353585b6db21e392c4318b8cff897738159feff562/respx-0.20.2-py2.py3-none-any.whl", hash = "sha256:ab8e1cf6da28a5b2dd883ea617f8130f77f676736e6e9e4a25817ad116a172c9", size = 22849, upload-time = "2023-07-20T23:01:21.994Z" }, -] - [[package]] name = "respx" version = "0.22.0" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] dependencies = [ - { name = "httpx", version = "0.28.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.8'" }, + { name = "httpx" }, ] sdist = { url = "https://files.pythonhosted.org/packages/f4/7c/96bd0bc759cf009675ad1ee1f96535edcb11e9666b985717eb8c87192a95/respx-0.22.0.tar.gz", hash = "sha256:3c8924caa2a50bd71aefc07aa812f2466ff489f1848c96e954a5362d17095d91", size = 28439, upload-time = "2024-12-19T22:33:59.374Z" } wheels = [ @@ -1402,24 +1017,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, ] -[[package]] -name = "tokenize-rt" -version = "5.0.0" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/40/01/fb40ea8c465f680bf7aa3f5bee39c62ba8b7f52c38048c27aa95aff4f779/tokenize_rt-5.0.0.tar.gz", hash = "sha256:3160bc0c3e8491312d0485171dea861fc160a240f5f5766b72a1165408d10740", size = 5329, upload-time = "2022-10-03T23:28:00.861Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/8d/12/4c7495f25b4c9131706f3aaffb185d4de32c02a6ee49d875e929c5b7c919/tokenize_rt-5.0.0-py2.py3-none-any.whl", hash = "sha256:c67772c662c6b3dc65edf66808577968fb10badfc2042e3027196bed4daf9e5a", size = 5848, upload-time = "2022-10-03T23:27:59.459Z" }, -] - [[package]] name = "tokenize-rt" version = "6.0.0" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/7d/09/6257dabdeab5097d72c5d874f29b33cd667ec411af6667922d84f85b79b5/tokenize_rt-6.0.0.tar.gz", hash = "sha256:b9711bdfc51210211137499b5e355d3de5ec88a85d2025c520cbb921b5194367", size = 5360, upload-time = "2024-08-04T21:01:19.405Z" } wheels = [ @@ -1448,27 +1051,10 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b", size = 16588, upload-time = "2020-11-01T01:40:20.672Z" }, ] -[[package]] -name = "tomli" -version = "2.0.1" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/c0/3f/d7af728f075fb08564c5949a9c95e44352e23dee646869fa104a3b2060a3/tomli-2.0.1.tar.gz", hash = "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f", size = 15164, upload-time = "2022-02-08T10:54:04.006Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl", hash = "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc", size = 12757, upload-time = "2022-02-08T10:54:02.017Z" }, -] - [[package]] name = "tomli" version = "2.3.0" source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.10'", - "python_full_version == '3.9.*'", - "python_full_version == '3.8.*'", -] sdist = { url = "https://files.pythonhosted.org/packages/52/ed/3f73f72945444548f33eba9a87fc7a6e969915e7b1acc8260b30e1f76a2f/tomli-2.3.0.tar.gz", hash = "sha256:64be704a875d2a59753d80ee8a533c3fe183e3f06807ff7dc2232938ccb01549", size = 17392, upload-time = "2025-10-08T22:01:47.119Z" } wheels = [ { url = "https://files.pythonhosted.org/packages/b3/2e/299f62b401438d5fe1624119c723f5d877acc86a4c2492da405626665f12/tomli-2.3.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:88bd15eb972f3664f5ed4b57c1634a97153b4bac4479dcb6a495f41921eb7f45", size = 153236, upload-time = "2025-10-08T22:01:00.137Z" }, @@ -1514,24 +1100,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/77/b8/0135fadc89e73be292b473cb820b4f5a08197779206b33191e801feeae40/tomli-2.3.0-py3-none-any.whl", hash = "sha256:e95b1af3c5b07d9e643909b5abbec77cd9f1217e6d0bca72b0234736b9fb1f1b", size = 14408, upload-time = "2025-10-08T22:01:46.04Z" }, ] -[[package]] -name = "typing-extensions" -version = "4.7.1" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/3c/8b/0111dd7d6c1478bf83baa1cab85c686426c7a6274119aceb2bd9d35395ad/typing_extensions-4.7.1.tar.gz", hash = "sha256:b75ddc264f0ba5615db7ba217daeb99701ad295353c45f9e95963337ceeeffb2", size = 72876, upload-time = "2023-07-02T14:20:55.045Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ec/6b/63cc3df74987c36fe26157ee12e09e8f9db4de771e0f3404263117e75b95/typing_extensions-4.7.1-py3-none-any.whl", hash = "sha256:440d5dd3af93b060174bf433bccd69b0babc3b15b1a8dca43789fd7f61514b36", size = 33232, upload-time = "2023-07-02T14:20:53.275Z" }, -] - [[package]] name = "typing-extensions" version = "4.13.2" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/f6/37/23083fcd6e35492953e8d2aaaa68b860eb422b34627b13f2ce3eb6106061/typing_extensions-4.13.2.tar.gz", hash = "sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef", size = 106967, upload-time = "2025-04-10T14:19:05.416Z" } wheels = [ @@ -1560,92 +1134,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/6f/81/5aace343f579809a50e152eac66458ad159c4fae9e06bef438ddb0ebaf92/vcdiff_decoder-0.2.0-py3-none-any.whl", hash = "sha256:0be7163871d0c79fb57b0c9f0a994ee151c4e348a442d3d508da6975983b91aa", size = 19277, upload-time = "2026-09-23T13:05:57.346Z" }, ] -[[package]] -name = "websockets" -version = "11.0.3" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/d8/3b/2ed38e52eed4cf277f9df5f0463a99199a04d9e29c9e227cfafa57bd3993/websockets-11.0.3.tar.gz", hash = "sha256:88fc51d9a26b10fc331be344f1781224a375b78488fc343620184e95a4b27016", size = 104235, upload-time = "2023-05-07T14:25:20.083Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/e1/76/88640f8aeac7eb0d058b913e7bb72682f8d569db44c7d30e576ec4777ce1/websockets-11.0.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:3ccc8a0c387629aec40f2fc9fdcb4b9d5431954f934da3eaf16cdc94f67dbfac", size = 123714, upload-time = "2023-05-07T14:23:15.309Z" }, - { url = "https://files.pythonhosted.org/packages/b9/6b/26b28115b46e23e74ede76d95792eedfe8c58b21f4daabfff1e9f159c8fe/websockets-11.0.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d67ac60a307f760c6e65dad586f556dde58e683fab03323221a4e530ead6f74d", size = 120949, upload-time = "2023-05-07T14:23:17.656Z" }, - { url = "https://files.pythonhosted.org/packages/f3/82/2d1f3395d47fab65fa8b801e2251b324300ed8db54753b6fb7919cef0c11/websockets-11.0.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:84d27a4832cc1a0ee07cdcf2b0629a8a72db73f4cf6de6f0904f6661227f256f", size = 121032, upload-time = "2023-05-07T14:23:20.096Z" }, - { url = "https://files.pythonhosted.org/packages/b2/ec/56bdd12d847e4fc2d0a7ba2d7f1476f79cda50599d11ffb6080b86f21ef1/websockets-11.0.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffd7dcaf744f25f82190856bc26ed81721508fc5cbf2a330751e135ff1283564", size = 130620, upload-time = "2023-05-07T14:23:21.545Z" }, - { url = "https://files.pythonhosted.org/packages/c9/fb/ae5ed4be3514287cf8f6c348c87e1392a6e3f4d6eadae75c18847a2f84b6/websockets-11.0.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7622a89d696fc87af8e8d280d9b421db5133ef5b29d3f7a1ce9f1a7bf7fcfa11", size = 129628, upload-time = "2023-05-07T14:23:23.105Z" }, - { url = "https://files.pythonhosted.org/packages/58/0a/7570e15661a0a546c3a1152d95fe8c05480459bab36247f0acbf41f01a41/websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bceab846bac555aff6427d060f2fcfff71042dba6f5fca7dc4f75cac815e57ca", size = 129938, upload-time = "2023-05-07T14:23:24.959Z" }, - { url = "https://files.pythonhosted.org/packages/ba/6c/5c0322b2875e8395e6bf0eff11f43f3e25da7ef5b12f4d908cd3a19ea841/websockets-11.0.3-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:54c6e5b3d3a8936a4ab6870d46bdd6ec500ad62bde9e44462c32d18f1e9a8e54", size = 134663, upload-time = "2023-05-07T14:23:26.382Z" }, - { url = "https://files.pythonhosted.org/packages/de/0e/d7274e4d41d7b34f204744c27a23707be2ecefaf6f7df2145655f086ecd7/websockets-11.0.3-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:41f696ba95cd92dc047e46b41b26dd24518384749ed0d99bea0a941ca87404c4", size = 133900, upload-time = "2023-05-07T14:23:28.307Z" }, - { url = "https://files.pythonhosted.org/packages/82/3c/00f051abcf88aec5e952a8840076749b0b26a30c219dcae8ba70200998aa/websockets-11.0.3-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:86d2a77fd490ae3ff6fae1c6ceaecad063d3cc2320b44377efdde79880e11526", size = 134520, upload-time = "2023-05-07T14:23:30.734Z" }, - { url = "https://files.pythonhosted.org/packages/8f/7b/4d4ecd29be7d08486e38f987a6603c491296d1e33fe55127d79aebb0333e/websockets-11.0.3-cp310-cp310-win32.whl", hash = "sha256:2d903ad4419f5b472de90cd2d40384573b25da71e33519a67797de17ef849b69", size = 124152, upload-time = "2023-05-07T14:23:33.183Z" }, - { url = "https://files.pythonhosted.org/packages/98/a7/0ed69892981351e5acf88fac0ff4c801fabca2c3bdef9fca4c7d3fde8c53/websockets-11.0.3-cp310-cp310-win_amd64.whl", hash = "sha256:1d2256283fa4b7f4c7d7d3e84dc2ece74d341bce57d5b9bf385df109c2a1a82f", size = 124674, upload-time = "2023-05-07T14:23:35.331Z" }, - { url = "https://files.pythonhosted.org/packages/16/49/ae616bd221efba84a3d78737b417f704af1ffa36f40dcaba5eb954dd4753/websockets-11.0.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:e848f46a58b9fcf3d06061d17be388caf70ea5b8cc3466251963c8345e13f7eb", size = 123748, upload-time = "2023-05-07T14:23:37.977Z" }, - { url = "https://files.pythonhosted.org/packages/0a/84/68b848a373493b58615d6c10e9e8ccbaadfd540f84905421739a807704f8/websockets-11.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:aa5003845cdd21ac0dc6c9bf661c5beddd01116f6eb9eb3c8e272353d45b3288", size = 120975, upload-time = "2023-05-07T14:23:40.339Z" }, - { url = "https://files.pythonhosted.org/packages/8c/a8/e81533499f84ef6cdd95d11d5b05fa827c0f097925afd86f16e6a2631d8e/websockets-11.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:b58cbf0697721120866820b89f93659abc31c1e876bf20d0b3d03cef14faf84d", size = 121017, upload-time = "2023-05-07T14:23:41.874Z" }, - { url = "https://files.pythonhosted.org/packages/6b/ca/65d6986665888494eca4d5435a9741c822022996f0f4200c57ce4b9242f7/websockets-11.0.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:660e2d9068d2bedc0912af508f30bbeb505bbbf9774d98def45f68278cea20d3", size = 131200, upload-time = "2023-05-07T14:23:43.309Z" }, - { url = "https://files.pythonhosted.org/packages/c0/a8/a8a582ebeeecc8b5f332997d44c57e241748f8a9856e06a38a5a13b30796/websockets-11.0.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c1f0524f203e3bd35149f12157438f406eff2e4fb30f71221c8a5eceb3617b6b", size = 130195, upload-time = "2023-05-07T14:23:45.337Z" }, - { url = "https://files.pythonhosted.org/packages/a9/5e/b25c60067d700e811dccb4e3c318eeadd3a19d8b3620de9f97434af777a7/websockets-11.0.3-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:def07915168ac8f7853812cc593c71185a16216e9e4fa886358a17ed0fd9fcf6", size = 130569, upload-time = "2023-05-07T14:23:46.926Z" }, - { url = "https://files.pythonhosted.org/packages/14/fc/5cbbf439c925e1e184a0392ec477a30cee2fabc0e63807c1d4b6d570fb52/websockets-11.0.3-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:b30c6590146e53149f04e85a6e4fcae068df4289e31e4aee1fdf56a0dead8f97", size = 136015, upload-time = "2023-05-07T14:23:48.43Z" }, - { url = "https://files.pythonhosted.org/packages/0f/d8/a997d3546aef9cc995a1126f7d7ade96c0e16c1a0efb9d2d430aee57c925/websockets-11.0.3-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:619d9f06372b3a42bc29d0cd0354c9bb9fb39c2cbc1a9c5025b4538738dbffaf", size = 135292, upload-time = "2023-05-07T14:23:50.744Z" }, - { url = "https://files.pythonhosted.org/packages/89/8f/707a05d5725f956c78d252a5fd73b89fa3ac57dd3959381c2d1acb41cb13/websockets-11.0.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:01f5567d9cf6f502d655151645d4e8b72b453413d3819d2b6f1185abc23e82dd", size = 135890, upload-time = "2023-05-07T14:23:52.707Z" }, - { url = "https://files.pythonhosted.org/packages/b5/94/ac47552208583d5dbcce468430c1eb2ae18962f6b3a694a2b7727cc60d4a/websockets-11.0.3-cp311-cp311-win32.whl", hash = "sha256:e1459677e5d12be8bbc7584c35b992eea142911a6236a3278b9b5ce3326f282c", size = 124149, upload-time = "2023-05-07T14:23:53.848Z" }, - { url = "https://files.pythonhosted.org/packages/e1/7c/0ad6e7ef0a054d73092f616d20d3d9bd3e1b837554cb20a52d8dd9f5b049/websockets-11.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:e7837cb169eca3b3ae94cc5787c4fed99eef74c0ab9506756eea335e0d6f3ed8", size = 124670, upload-time = "2023-05-07T14:23:55.812Z" }, - { url = "https://files.pythonhosted.org/packages/b5/a8/8900184ab0b06b6e620ba7e92cf2faa5caa9ba86e148541b8fff1c7b6646/websockets-11.0.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:9f59a3c656fef341a99e3d63189852be7084c0e54b75734cde571182c087b152", size = 120868, upload-time = "2023-05-07T14:23:57.24Z" }, - { url = "https://files.pythonhosted.org/packages/44/a8/66c3a66b70b01a6c55fde486298766177fa11dd0d3a2c1cfc6820f25b4dc/websockets-11.0.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2529338a6ff0eb0b50c7be33dc3d0e456381157a31eefc561771ee431134a97f", size = 130557, upload-time = "2023-05-07T14:23:59.274Z" }, - { url = "https://files.pythonhosted.org/packages/70/fc/71377f36ef3049f3bc7db7c0f3a7696929d5f836d7a18777131d994192a9/websockets-11.0.3-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:34fd59a4ac42dff6d4681d8843217137f6bc85ed29722f2f7222bd619d15e95b", size = 129640, upload-time = "2023-05-07T14:24:01.412Z" }, - { url = "https://files.pythonhosted.org/packages/36/19/0da435afb26a6c47c0c045a82e414912aa2ac10de5721276a342bd9fdfee/websockets-11.0.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:332d126167ddddec94597c2365537baf9ff62dfcc9db4266f263d455f2f031cb", size = 129903, upload-time = "2023-05-07T14:24:02.872Z" }, - { url = "https://files.pythonhosted.org/packages/38/ed/b8b133416536b6816e480594864e5950051db522714623eefc9e5275ec04/websockets-11.0.3-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:6505c1b31274723ccaf5f515c1824a4ad2f0d191cec942666b3d0f3aa4cb4007", size = 135302, upload-time = "2023-05-07T14:24:04.326Z" }, - { url = "https://files.pythonhosted.org/packages/e9/26/1dfaa81788f61c485b4d65f1b28a19615e39f9c45100dce5e2cbf5ad1352/websockets-11.0.3-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:f467ba0050b7de85016b43f5a22b46383ef004c4f672148a8abf32bc999a87f0", size = 134562, upload-time = "2023-05-07T14:24:05.829Z" }, - { url = "https://files.pythonhosted.org/packages/a7/f7/1e852351e8073c32885172a6bef64c95d14c13ff3634b01d4a1086321491/websockets-11.0.3-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:9d9acd80072abcc98bd2c86c3c9cd4ac2347b5a5a0cae7ed5c0ee5675f86d9af", size = 135191, upload-time = "2023-05-07T14:24:07.659Z" }, - { url = "https://files.pythonhosted.org/packages/19/d3/2ea3f95d83033675144b0848a0ae2e4998b3f763da09ec3df6bce97ea4e6/websockets-11.0.3-cp37-cp37m-win32.whl", hash = "sha256:e590228200fcfc7e9109509e4d9125eace2042fd52b595dd22bbc34bb282307f", size = 124138, upload-time = "2023-05-07T14:24:09.697Z" }, - { url = "https://files.pythonhosted.org/packages/94/8c/266155c14b7a26deca6fa4c4d5fd15b0ab32725d78a2acfcf6b24943585d/websockets-11.0.3-cp37-cp37m-win_amd64.whl", hash = "sha256:b16fff62b45eccb9c7abb18e60e7e446998093cdcb50fed33134b9b6878836de", size = 124672, upload-time = "2023-05-07T14:24:12.595Z" }, - { url = "https://files.pythonhosted.org/packages/c4/5b/60eccd7e9703bbe93fc4167d1e7ada7e8e8e51544122198d63fd8e3460b7/websockets-11.0.3-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:fb06eea71a00a7af0ae6aefbb932fb8a7df3cb390cc217d51a9ad7343de1b8d0", size = 123706, upload-time = "2023-05-07T14:24:14.633Z" }, - { url = "https://files.pythonhosted.org/packages/d1/ec/7e2b9bebc2e9b4a48404144106bbc6a7ace781feeb0e6a3829551e725fa5/websockets-11.0.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:8a34e13a62a59c871064dfd8ffb150867e54291e46d4a7cf11d02c94a5275bae", size = 120944, upload-time = "2023-05-07T14:24:16.144Z" }, - { url = "https://files.pythonhosted.org/packages/8a/bd/a5e5973899d78d44a540f50a9e30b01c6771e8bf7883204ee762060cf95a/websockets-11.0.3-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:4841ed00f1026dfbced6fca7d963c4e7043aa832648671b5138008dc5a8f6d99", size = 121030, upload-time = "2023-05-07T14:24:17.905Z" }, - { url = "https://files.pythonhosted.org/packages/ec/3f/0c5cae14e9e86401105833383405787ae4caddd476a8fc5561259253dab7/websockets-11.0.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1a073fc9ab1c8aff37c99f11f1641e16da517770e31a37265d2755282a5d28aa", size = 130811, upload-time = "2023-05-07T14:24:21.175Z" }, - { url = "https://files.pythonhosted.org/packages/a0/39/acc3d4b15c5207ef7cca823c37eca8c74e3e1a1a63a397798986be3bdef7/websockets-11.0.3-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:68b977f21ce443d6d378dbd5ca38621755f2063d6fdb3335bda981d552cfff86", size = 129876, upload-time = "2023-05-07T14:24:22.498Z" }, - { url = "https://files.pythonhosted.org/packages/58/05/2efb520317340ece74bfc4d88e8f011dd71a4e6c263000bfffb71a343685/websockets-11.0.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e1a99a7a71631f0efe727c10edfba09ea6bee4166a6f9c19aafb6c0b5917d09c", size = 130158, upload-time = "2023-05-07T14:24:25.193Z" }, - { url = "https://files.pythonhosted.org/packages/30/a5/d641f2a9a4b4079cfddbb0726fc1b914be76a610aaedb45e4760899a4ce1/websockets-11.0.3-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:bee9fcb41db2a23bed96c6b6ead6489702c12334ea20a297aa095ce6d31370d0", size = 134494, upload-time = "2023-05-07T14:24:26.463Z" }, - { url = "https://files.pythonhosted.org/packages/ca/20/25211be61d50189650fb0ec6084b6d6339f5c7c6436a6c217608dcb553e4/websockets-11.0.3-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:4b253869ea05a5a073ebfdcb5cb3b0266a57c3764cf6fe114e4cd90f4bfa5f5e", size = 133735, upload-time = "2023-05-07T14:24:29.598Z" }, - { url = "https://files.pythonhosted.org/packages/c6/91/f36454b87edf10a95be9c7212d2dcb8c606ddbf7a183afdc498933acdd19/websockets-11.0.3-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:1553cb82942b2a74dd9b15a018dce645d4e68674de2ca31ff13ebc2d9f283788", size = 134368, upload-time = "2023-05-07T14:24:30.923Z" }, - { url = "https://files.pythonhosted.org/packages/58/68/9403771de1b1c21a2e878e4841815af8c9f8893b094654934e2a5ee4dbc8/websockets-11.0.3-cp38-cp38-win32.whl", hash = "sha256:f61bdb1df43dc9c131791fbc2355535f9024b9a04398d3bd0684fc16ab07df74", size = 124148, upload-time = "2023-05-07T14:24:32.299Z" }, - { url = "https://files.pythonhosted.org/packages/25/25/48540419005d07ed2d368a7eafb44ed4f33a2691ae4c210850bf31123c4a/websockets-11.0.3-cp38-cp38-win_amd64.whl", hash = "sha256:03aae4edc0b1c68498f41a6772d80ac7c1e33c06c6ffa2ac1c27a07653e79d6f", size = 124665, upload-time = "2023-05-07T14:24:34.482Z" }, - { url = "https://files.pythonhosted.org/packages/c0/21/cb9dfbbea8dc0ad89ced52630e7e61edb425fb9fdc6002f8d0c5dd26b94b/websockets-11.0.3-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:777354ee16f02f643a4c7f2b3eff8027a33c9861edc691a2003531f5da4f6bc8", size = 123707, upload-time = "2023-05-07T14:24:36.007Z" }, - { url = "https://files.pythonhosted.org/packages/8f/f2/8a3eb016be19743c7eb9e67c855df0fdfa5912534ffaf83a05b62667d761/websockets-11.0.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:8c82f11964f010053e13daafdc7154ce7385ecc538989a354ccc7067fd7028fd", size = 120963, upload-time = "2023-05-07T14:24:37.478Z" }, - { url = "https://files.pythonhosted.org/packages/a0/1a/3da73e69ebc00649d11ed836541c92c1a2df0b8a8aa641a2c8746e7c2b9c/websockets-11.0.3-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:3580dd9c1ad0701169e4d6fc41e878ffe05e6bdcaf3c412f9d559389d0c9e016", size = 121014, upload-time = "2023-05-07T14:24:39.009Z" }, - { url = "https://files.pythonhosted.org/packages/d9/36/5741e62ccf629c8e38cc20f930491f8a33ce7dba972cae93dba3d6f02552/websockets-11.0.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6f1a3f10f836fab6ca6efa97bb952300b20ae56b409414ca85bff2ad241d2a61", size = 130408, upload-time = "2023-05-07T14:24:40.825Z" }, - { url = "https://files.pythonhosted.org/packages/66/89/799f595c67b97a8a17e13d2764e088f631616bd95668aaa4c04b7cada136/websockets-11.0.3-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:df41b9bc27c2c25b486bae7cf42fccdc52ff181c8c387bfd026624a491c2671b", size = 129407, upload-time = "2023-05-07T14:24:42.479Z" }, - { url = "https://files.pythonhosted.org/packages/a6/9c/2356ecb952fd3992b73f7a897d65e57d784a69b94bb8d8fd5f97531e5c02/websockets-11.0.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:279e5de4671e79a9ac877427f4ac4ce93751b8823f276b681d04b2156713b9dd", size = 129712, upload-time = "2023-05-07T14:24:44.186Z" }, - { url = "https://files.pythonhosted.org/packages/c4/f5/15998b164c183af0513bba744b51ecb08d396ff86c0db3b55d62624d1f15/websockets-11.0.3-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:1fdf26fa8a6a592f8f9235285b8affa72748dc12e964a5518c6c5e8f916716f7", size = 134386, upload-time = "2023-05-07T14:24:45.702Z" }, - { url = "https://files.pythonhosted.org/packages/8a/77/a04d2911f6e2b9e781ce7ffc1e8516b54b85f985369eec8c853fd619d8e8/websockets-11.0.3-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:69269f3a0b472e91125b503d3c0b3566bda26da0a3261c49f0027eb6075086d1", size = 133639, upload-time = "2023-05-07T14:24:46.966Z" }, - { url = "https://files.pythonhosted.org/packages/72/89/0d150939f2e592ed78c071d69237ac1c872462cc62a750c5f592f3d4ab18/websockets-11.0.3-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:97b52894d948d2f6ea480171a27122d77af14ced35f62e5c892ca2fae9344311", size = 134260, upload-time = "2023-05-07T14:24:48.633Z" }, - { url = "https://files.pythonhosted.org/packages/9a/6e/0fd7274042f46acb589161407f4b505b44c68d369437ce919bae1fa9b8c4/websockets-11.0.3-cp39-cp39-win32.whl", hash = "sha256:c7f3cb904cce8e1be667c7e6fef4516b98d1a6a0635a58a57528d577ac18a128", size = 124146, upload-time = "2023-05-07T14:24:50.566Z" }, - { url = "https://files.pythonhosted.org/packages/f4/3f/65dfa50084a06ab0a05f3ca74195c2c17a1c075b8361327d831ccce0a483/websockets-11.0.3-cp39-cp39-win_amd64.whl", hash = "sha256:c792ea4eabc0159535608fc5658a74d1a81020eb35195dd63214dcf07556f67e", size = 124665, upload-time = "2023-05-07T14:24:52.111Z" }, - { url = "https://files.pythonhosted.org/packages/e2/2f/3ad8ac4a9dc9d685e098e534180a36ed68fe2e85e82e225e00daec86bb94/websockets-11.0.3-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:f2e58f2c36cc52d41f2659e4c0cbf7353e28c8c9e63e30d8c6d3494dc9fdedcf", size = 120795, upload-time = "2023-05-07T14:24:53.421Z" }, - { url = "https://files.pythonhosted.org/packages/a7/8c/7100e9cf310fe1d83d1ae1322203f4eb2b767a7c2b301c1e70db6270306f/websockets-11.0.3-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:de36fe9c02995c7e6ae6efe2e205816f5f00c22fd1fbf343d4d18c3d5ceac2f5", size = 122910, upload-time = "2023-05-07T14:24:54.851Z" }, - { url = "https://files.pythonhosted.org/packages/78/b2/df5452031b02b857851139806308f2af7c749069e25bfe15f2d559ade6e7/websockets-11.0.3-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0ac56b661e60edd453585f4bd68eb6a29ae25b5184fd5ba51e97652580458998", size = 122516, upload-time = "2023-05-07T14:24:56.477Z" }, - { url = "https://files.pythonhosted.org/packages/03/28/3a51ffcf51ac45746639f83128908bbb1cd212aa631e42d15a7acebce5cb/websockets-11.0.3-pp37-pypy37_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e052b8467dd07d4943936009f46ae5ce7b908ddcac3fda581656b1b19c083d9b", size = 122462, upload-time = "2023-05-07T14:24:57.77Z" }, - { url = "https://files.pythonhosted.org/packages/eb/fb/2af7fc3ce2c3f1378d48a15802b4ff2caf6c0dfac13291e73c557caf04f7/websockets-11.0.3-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:42cc5452a54a8e46a032521d7365da775823e21bfba2895fb7b77633cce031bb", size = 124704, upload-time = "2023-05-07T14:24:59.322Z" }, - { url = "https://files.pythonhosted.org/packages/20/62/5c6039c4069912adb27889ddd000403a2de9e0fe6aebe439b4e6b128a6b8/websockets-11.0.3-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:e6316827e3e79b7b8e7d8e3b08f4e331af91a48e794d5d8b099928b6f0b85f20", size = 120795, upload-time = "2023-05-07T14:25:01.047Z" }, - { url = "https://files.pythonhosted.org/packages/38/30/01a10fbf4cc1e7ffa07be9b0401501918fc9433d71fb7da4cfcef3bd26ca/websockets-11.0.3-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8531fdcad636d82c517b26a448dcfe62f720e1922b33c81ce695d0edb91eb931", size = 122908, upload-time = "2023-05-07T14:25:02.734Z" }, - { url = "https://files.pythonhosted.org/packages/99/23/43071c989c0f87f612e7bccee98d00b04bddd3aca0cdc1ffaf31f6f8a4b4/websockets-11.0.3-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c114e8da9b475739dde229fd3bc6b05a6537a88a578358bc8eb29b4030fac9c9", size = 122515, upload-time = "2023-05-07T14:25:04.803Z" }, - { url = "https://files.pythonhosted.org/packages/b6/96/0d586c25d043aeab9457dad8e407251e3baf314d871215f91847e7b995c4/websockets-11.0.3-pp38-pypy38_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e063b1865974611313a3849d43f2c3f5368093691349cf3c7c8f8f75ad7cb280", size = 122465, upload-time = "2023-05-07T14:25:06.352Z" }, - { url = "https://files.pythonhosted.org/packages/27/e9/605b0618d0864e9be7c2a78f22bff57aba9cf56b9fccde3205db9023ae22/websockets-11.0.3-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:92b2065d642bf8c0a82d59e59053dd2fdde64d4ed44efe4870fa816c1232647b", size = 124707, upload-time = "2023-05-07T14:25:07.782Z" }, - { url = "https://files.pythonhosted.org/packages/1b/3d/3dc77699fa4d003f2e810c321592f80f62b81d7b78483509de72ffe581fd/websockets-11.0.3-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:0ee68fe502f9031f19d495dae2c268830df2760c0524cbac5d759921ba8c8e82", size = 120795, upload-time = "2023-05-07T14:25:09.785Z" }, - { url = "https://files.pythonhosted.org/packages/a6/1b/5c83c40f8d3efaf0bb2fdf05af94fb920f74842b7aaf31d7598e3ee44d58/websockets-11.0.3-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:dcacf2c7a6c3a84e720d1bb2b543c675bf6c40e460300b628bab1b1efc7c034c", size = 122909, upload-time = "2023-05-07T14:25:11.243Z" }, - { url = "https://files.pythonhosted.org/packages/32/2c/ab8ea64e9a7d8bf62a7ea7a037fb8d328d8bd46dbfe083787a9d452a148e/websockets-11.0.3-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b67c6f5e5a401fc56394f191f00f9b3811fe843ee93f4a70df3c389d1adf857d", size = 122517, upload-time = "2023-05-07T14:25:12.881Z" }, - { url = "https://files.pythonhosted.org/packages/8b/97/34178f5f7c29e679372d597cebfeff2aa45991d741d938117d4616e81a74/websockets-11.0.3-pp39-pypy39_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1d5023a4b6a5b183dc838808087033ec5df77580485fc533e7dab2567851b0a4", size = 122463, upload-time = "2023-05-07T14:25:15.154Z" }, - { url = "https://files.pythonhosted.org/packages/ed/45/466944e00b324ae3a1fddb305b4abf641f582e131548f07bcd970971b154/websockets-11.0.3-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:ed058398f55163a79bb9f06a90ef9ccc063b204bb346c4de78efc5d15abfe602", size = 124707, upload-time = "2023-05-07T14:25:17.112Z" }, - { url = "https://files.pythonhosted.org/packages/47/96/9d5749106ff57629b54360664ae7eb9afd8302fad1680ead385383e33746/websockets-11.0.3-py3-none-any.whl", hash = "sha256:6681ba9e7f8f3b19440921e99efbb40fc89f26cd71bf539e45d8c8a25c976dc6", size = 118056, upload-time = "2023-05-07T14:25:18.508Z" }, -] - [[package]] name = "websockets" version = "13.1" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/e2/73/9223dbc7be3dcaf2a7bbf756c351ec8da04b1fa573edaf545b95f6b0c7fd/websockets-13.1.tar.gz", hash = "sha256:a3b3366087c1bc0a2795111edcadddb8b3b59509d5db5d7ea3fdd69f954a8878", size = 158549, upload-time = "2024-09-21T17:34:21.54Z" } wheels = [ @@ -1816,24 +1310,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" }, ] -[[package]] -name = "zipp" -version = "3.15.0" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.8'", -] -sdist = { url = "https://files.pythonhosted.org/packages/00/27/f0ac6b846684cecce1ee93d32450c45ab607f65c2e0255f0092032d91f07/zipp-3.15.0.tar.gz", hash = "sha256:112929ad649da941c23de50f356a2b5570c954b65150642bccdd66bf194d224b", size = 18454, upload-time = "2023-02-25T02:17:22.503Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/5b/fa/c9e82bbe1af6266adf08afb563905eb87cab83fde00a0a08963510621047/zipp-3.15.0-py3-none-any.whl", hash = "sha256:48904fc76a60e542af151aded95726c1a5c34ed43ab4134b597665c86d7ad556", size = 6758, upload-time = "2023-02-25T02:17:20.807Z" }, -] - [[package]] name = "zipp" version = "3.20.2" source = { registry = "https://pypi.org/simple" } resolution-markers = [ - "python_full_version == '3.8.*'", + "python_full_version < '3.9'", ] sdist = { url = "https://files.pythonhosted.org/packages/54/bf/5c0000c44ebc80123ecbdddba1f5dcd94a5ada602a9c225d84b5aaa55e86/zipp-3.20.2.tar.gz", hash = "sha256:bc9eb26f4506fda01b81bcde0ca78103b6e62f991b381fec825435c836edbc29", size = 24199, upload-time = "2024-09-13T13:44:16.101Z" } wheels = [