Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 18 additions & 97 deletions .rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,122 +2,43 @@

This file provides guidance to programming agents when working with code in this repository.

## Project Overview
Python client for the [Apify API](https://docs.apify.com/api/v2), published on PyPI as `apify-client`. `ApifyClient` (sync) and `ApifyClientAsync` (async) are the entry points.

Python client for the [Apify API](https://docs.apify.com/api/v2). Provides `ApifyClient` (sync) and `ApifyClientAsync` (async) classes to interact with the Apify platform. Published on PyPI as `apify-client`.
Read [CONTRIBUTING.md](CONTRIBUTING.md) first. It covers setup, tasks, tests, code style, generated models, documentation, and commits, and all of it applies to agents too.

## Development Commands

Uses [uv](https://docs.astral.sh/uv/) for project management and [Poe the Poet](https://poethepoet.natn.io/) as task runner. Requires Python 3.11+.

```bash
uv run poe install-dev # Install dev deps + git hooks
uv run poe check-code # Run all checks (lint, type-check, unit-tests, docstring check)
uv run poe lint # Ruff format check + ruff check
uv run poe format # Auto-fix lint issues and format
uv run poe type-check # Run ty type checker
uv run poe unit-tests # Run unit tests
uv run poe check-docstrings # Verify async docstrings match sync
uv run poe fix-docstrings # Auto-fix async docstrings
uv run poe generate-models # Regenerate _models.py, _typeddicts.py, _literals.py from the published spec
uv run poe generate-models-from-file <path> # Regenerate from a local OpenAPI spec file

# Run a single test
uv run pytest tests/unit/test_file.py
uv run pytest tests/unit/test_file.py::test_name
```

Integration tests require `APIFY_TEST_USER_API_TOKEN` and `APIFY_TEST_USER_2_API_TOKEN` env vars.
@CONTRIBUTING.md

## Architecture

### Client Hierarchy
### Client hierarchy

`ApifyClient`/`ApifyClientAsync` are the entry points. They provide methods that return resource sub-clients:
Entry points return resource sub-clients, which can create child clients (e.g. `ActorClient.builds()` -> `BuildCollectionClient`):

```text
ApifyClient
├── .actor(id) → ActorClient (single resource operations)
├── .actor(id) → ActorClient (single resource operations)
├── .actors() → ActorCollectionClient (list/create)
├── .dataset(id) → DatasetClient
├── .datasets() → DatasetCollectionClient
├── .run(id) → RunClient
├── .runs() → RunCollectionClient
└── ... (schedules, tasks, webhooks, key-value stores, request queues, etc.)
└── ... (datasets, schedules, tasks, webhooks, key-value stores, request queues, etc.)
```

Each resource client can create child clients (e.g., `ActorClient.builds()` → `BuildCollectionClient`).

### Sync/Async Symmetry

Every resource client has both sync and async variants in the **same file**. For example, `actor.py` contains both `ActorClient` and `ActorClientAsync`. Both share the same interface — async versions use `async/await`.

Docstrings are written on sync clients and **automatically copied** to async clients via `scripts/check_docstrings.py`. When modifying docstrings, edit only the sync client and run `uv run poe fix-docstrings`.
`ClientRegistry` / `ClientRegistryAsync` (`_client_registry.py`) hold the resource client classes and are passed to every client, so clients can create child clients without importing them (avoids circular imports). All resource clients derive from `ResourceClientBase` (`_resource_clients/_resource_client.py`), whose `WithLogDetailsClient` metaclass wraps public methods for structured logging.

### Dependency Injection via ClientRegistry
### Sync/async symmetry

`ClientRegistry`/`ClientRegistryAsync` (in `_client_registry.py`) holds references to all resource client classes and is passed to each client. This avoids circular imports and lets resource clients create child clients without directly importing them.
Every resource client in `src/apify_client/_resource_clients/` has its sync and async variant in the **same file** (e.g. `ActorClient` and `ActorClientAsync` in `actor.py`) with the same interface. Docstrings are written on the sync client only (see [Documentation](CONTRIBUTING.md#documentation)).

### HTTP Client Abstraction
### HTTP layer

- `HttpClient`/`HttpClientAsync` — base classes in `http_clients/_base.py` holding the shared request pipeline (retries,
timeouts, API errors); transports implement the `send_request`, error-classification, and lifecycle hooks
- `ImpitHttpClient`/`ImpitHttpClientAsync` — default implementation (Rust-based Impit)
- `Httpx2HttpClient`/`Httpx2HttpClientAsync` — built-in alternative behind the `httpx2` optional extra
- `HttpResponse` — Protocol (not a concrete class) for response objects
- Users can plug in custom HTTP clients via `ApifyClient.with_custom_http_client()`
`HttpClient` / `HttpClientAsync` (`http_clients/_base.py`) hold the shared request pipeline (retries, timeouts, API errors); transports (`Impit*` default, `Httpx2*` behind the `httpx2` extra, or a custom one via `ApifyClient.with_custom_http_client()`) implement sending, error classification, and lifecycle hooks. Request-body compression lives in `http_compressors/`, selected by the `compression` constructor argument.

### Base Classes
### Generated models

- `ResourceClientBase` — shared URL building, parameter handling, metaclass-based logging
- `ResourceClient` / `ResourceClientAsync` — sync/async base classes with HTTP call methods
- `WithLogDetailsClient` metaclass — auto-wraps public methods for structured logging
`_models.py`, `_typeddicts.py`, and `_literals.py` are **generated**; never edit them by hand (see [Generated models](CONTRIBUTING.md#generated-models)). Each input-side TypedDict ships in a snake_case and a camelCase variant (`RequestResourceDict`, `RequestResourceCamelDict`); the camel variants are synthesized in the post-processing script, and resource-client signatures accept either casing or the Pydantic model.

### Data Models
### Checking API response shapes

`src/apify_client/_models.py` and `src/apify_client/_typeddicts.py` are **auto-generated** — do not edit them manually. Every Pydantic model and TypedDict comes from the OpenAPI spec.
The OpenAPI spec can lag behind the API and omit fields, and a live response only shows the fields that happen to be set. If the private `apify-core` repository is available locally, use it to confirm which fields an endpoint returns:

Each input-side TypedDict ships in two casings: snake_case (`RequestResourceDict`) and camelCase (`RequestResourceCamelDict`). The camel variants are synthesized in `scripts/postprocess_generated_models.py` from the Pydantic `Field(alias=...)` map; resource-client signatures union both with the Pydantic model so users can pass either casing without losing type-checker support.

- Generated by `datamodel-code-generator` from the OpenAPI spec at `https://docs.apify.com/api/openapi.json` (config in `pyproject.toml` under `[tool.datamodel-codegen]`, aliases in `datamodel_codegen_aliases.json`)
- After generation, `scripts/postprocess_generated_models.py` is run to apply additional fixes
- To regenerate locally:
- From the live published spec: `uv run poe generate-models`
- From a local spec file: `uv run poe generate-models-from-file path/to/openapi.json`
- `scripts/openapi_spec.py` downloads the spec once into git-ignored `tmp/openapi.json` — both codegen passes read that copy, so a spec redeployed mid-run can't leave the generated files describing two different inputs — then records its `info.version` in `pyproject.toml` under `[tool.apify.openapi-spec]`
- That version is a **coarse marker, not a content identity**: the spec is served latest-only, and apify-docs bumps `components/version.yaml` in a follow-up `[skip ci]` commit, so a deploy can publish new content under the old stamp. A moved value proves the spec changed; an unchanged value proves nothing
- Recording is unconditional; the nightly workflow keeps it honest by committing nothing unless the models changed. A local `generate-models` that only moves that line should be dropped, not committed alone. `generate-models-from-file` leaves the line alone, so models from a candidate spec must not be committed
- In CI, the `Regenerate models` workflow (`on_schedule_regenerate_models.yaml`) runs nightly at 02:00 UTC and opens a PR when the models change. It always generates on master and rebuilds its `ci/regenerate-models` branch from master instead of appending, so the PR diff is always "current spec vs current master". Retitle the `chore:` PR to `fix:`/`feat:` when the diff is user-facing, so it lands in the changelog and triggers a release
- The gate is the **generated models**, not the spec version — which moves on rebuilds regardless of client impact, and doesn't reliably move when the content does
- Manual regeneration is also possible from the GitHub Actions UI (`Regenerate models` workflow)

## Code Conventions

- **Line length**: 120 characters
- **Linting/formatting**: Ruff with nearly all rules enabled (see `pyproject.toml`)
- **Type checking**: ty (Astral's type checker)
- **Docstrings**: Google style format, single backticks for inline code references (`` `ApifyClient` `` not ``` ``ApifyClient`` ```)
- **Imports**: `from __future__ import annotations` used throughout
- **Commits**: [Conventional Commits](https://www.conventionalcommits.org/) format. Choose the type based on *what* changed, not just *why*:
- `feat:` / `fix:` / `perf:` / `refactor:` / `style:` — **source code only**; these trigger a release and appear in the changelog
- `test:` — test additions or changes (no release triggered)
- `docs:` — documentation changes; also triggers a doc release on master
- `ci:` — CI/workflow changes
- `chore:` — dependency bumps, tooling, generated files (e.g. model regeneration PRs), and other housekeeping
- `build:` — build system changes

## Testing

- **Unit tests** (`tests/unit/`): Use `pytest-httpserver` to mock HTTP. No network required.
- **Integration tests** (`tests/integration/`): Hit the live Apify API.
- **Async**: `asyncio_mode = "auto"` — async tests run automatically without markers.
- **Parallelism**: Tests run in parallel via `pytest-xdist`.

Unit test pattern:

```python
def test_example(httpserver: HTTPServer) -> None:
httpserver.expect_request('/v2/endpoint').respond_with_json({'data': ...})
client = ApifyClient(token='test', api_url=httpserver.url_for('/').removesuffix('/'))
# assert client behavior
```
- A single resource's fields are in `PUBLISHED_*_FIELDS` in `src/packages/<area>-server/src/<resource>/<resource>.server.ts`.
- A list endpoint has its own, usually smaller, `API_PUBLISHED_*_LIST_FIELDS` in `src/api/src/routes/<resource>/<resource>_list.ts`, and often builds each item inline. Don't assume a list item has the same fields as the single resource.
Loading
Loading