feat: sync .NET client with Apify OpenAPI spec v2-2026-10-19T133704Z

CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## 0.6.0
+
+- Bumped `ApifyClientVersion.ApiSpecVersion` to the Apify OpenAPI spec `v2-2026-10-19T133704Z` and
+  the project version to `0.6.0`.
+- Added `ApifyClientException`, a common base class that `ApifyApiException` (and its per-status
+  subclasses) and `ApifyTransportException` now extend, matching the reference client's
+  `ApifyClientError`. A single `catch (ApifyClientException)` now catches every error this client
+  raises for a failed call.
+- Added `ContentEncoding` to `SetRecordOptions`, for uploading a key-value-store record that is already
+  compressed (`gzip`, `br`, `deflate`): the client sends the bytes as-is and skips its own compression,
+  matching the reference client's `setRecord` `contentEncoding` option.
+- Added `DatasetClient.StreamItemsAsync`, which downloads a dataset item export as a `Stream` instead
+  of buffering it into a `byte[]`, matching the reference client's `downloadItems` `stream` option.
+- Added `LogClient.GetBytesAsync`, which fetches a log as raw bytes instead of a decoded string,
+  matching the reference client's `log.get` `buffer` option.
+- Added `KeyValueStoreClient.IterateKeysAsync` (with a new `IterateKeysOptions`, whose `MaxPageLimit`
+  caps the per-page size), an auto-paging iterator over a store's keys, matching the reference client's
+  chunked `listKeys()` iteration. `Limit = 0` yields no keys without issuing any request (the API itself
+  reads a literal `limit=0` as unbounded, not zero); `Limit` must not be negative and `MaxPageLimit` must
+  be positive. Pages continue only while the API returns a non-null `nextExclusiveStartKey`, matching the
+  reference client, rather than on `isTruncated` alone.
+
 ## 0.5.0
 
 - Bumped `ApifyClientVersion.ApiSpecVersion` to the Apify OpenAPI spec `v2-2026-10-08T073338Z` and

docs/README.md

@@ -73,8 +73,8 @@ the `using` directives for whichever ones a file references:
 | `Apify.Client` | The entry point (`ApifyClient`), `ApifyClientOptions`, `ApifyClientVersion`, and the log-redirection helper `StreamedLog`. |
 | `Apify.Client.Resources` | Every resource client the entry point returns — `ActorClient`, `RunClient`, `BuildClient`, `DatasetClient`, `KeyValueStoreClient`, `RequestQueueClient`, `TaskClient`, `ScheduleClient`, `LogClient`, `UserClient`, `ActorVersionClient`, `ActorEnvVarClient`, the `…CollectionClient` types (including `ActorVersionCollectionClient`, `ActorEnvVarCollectionClient`, `NestedWebhookCollectionClient`, and `WebhookDispatchCollectionClient`), etc. |
 | `Apify.Client.Models` | Data models returned by the clients — `Actor`, `ActorRun`, `Build`, `Dataset`, `RequestQueueRequest`, `ActorEnvVar`, `PaginationList<T>`, and so on. |
-| `Apify.Client.Options` | The option/request objects passed into methods — `ActorStartOptions`, `ActorBuildOptions`, `ActorListOptions`, `RunListOptions`, `RunResurrectOptions`, `RunChargeOptions`, `MetamorphOptions`, `LastRunOptions`, `DatasetListItemsOptions`, `DatasetDownloadOptions`, `DownloadItemsFormat`, `GetRecordOptions`, `SetRecordOptions`, `ListKeysOptions`, `ListRequestsOptions`, `PaginateRequestsOptions`, `BatchAddRequestsOptions`, `RequestQueueClientOptions`, `TaskStartOptions`, `ValidateInputOptions`, `LogOptions`, `ListOptions`, `StorageListOptions`, `StoreListOptions`. |
-| `Apify.Client.Exceptions` | `ApifyApiException` and `ApifyTransportException`. |
+| `Apify.Client.Options` | The option/request objects passed into methods — `ActorStartOptions`, `ActorBuildOptions`, `ActorListOptions`, `RunListOptions`, `RunResurrectOptions`, `RunChargeOptions`, `MetamorphOptions`, `LastRunOptions`, `DatasetListItemsOptions`, `DatasetDownloadOptions`, `DownloadItemsFormat`, `GetRecordOptions`, `SetRecordOptions`, `ListKeysOptions`, `IterateKeysOptions`, `ListRequestsOptions`, `PaginateRequestsOptions`, `BatchAddRequestsOptions`, `RequestQueueClientOptions`, `TaskStartOptions`, `ValidateInputOptions`, `LogOptions`, `ListOptions`, `StorageListOptions`, `StoreListOptions`. |
+| `Apify.Client.Exceptions` | `ApifyClientException` (common base), `ApifyApiException` (and its per-status subclasses) and `ApifyTransportException`. |
 | `Apify.Client.Http` | The replaceable transport: `IHttpTransport` and the default `HttpClientTransport`. |
 | `System.Text.Json.Nodes` (BCL) | Not an Apify namespace, but required whenever you name the JSON escape-hatch types the client returns/accepts — `JsonObject`/`JsonNode` (e.g. dataset items, `GetInputAsync`/`UpdateInputAsync`, `GetStatisticsAsync`, `GetOpenApiDefinitionAsync`, `MonthlyUsageAsync`/`LimitsAsync`, `RequestQueueRequest.UserData`). Add `using System.Text.Json.Nodes;`. |
 
@@ -205,6 +205,17 @@ catch (ApifyApiException e)
 `ApifyTransportException` is thrown instead when the request never reaches the API (network failure,
 timeout, or DNS error) after all retries are exhausted.
 
+Both `ApifyApiException` and `ApifyTransportException` extend the common `ApifyClientException`, so
+`catch (ApifyClientException)` catches every error this client raises for a failed call without naming
+either concrete type:
+
+```csharp
+catch (ApifyClientException e)
+{
+    Console.WriteLine("client call failed: " + e.Message);
+}
+```
+
 ## Versioning
 
 - `Apify.Client.ApifyClientVersion.ClientVersion` — the semantic version of this library.

docs/misc.md

@@ -63,13 +63,15 @@ Console.WriteLine(otherUser?.Username);
 ## Logs — `client.Log(buildOrRunId)`
 
 - `GetAsync(LogOptions? options = null)` → `string?` (buffered; `null` if the log does not exist).
+- `GetBytesAsync(LogOptions? options = null)` → `byte[]?` — the same buffered fetch as `GetAsync`, as raw
+  bytes instead of a decoded string (for a log that is not necessarily valid UTF-8 text).
 - `StreamAsync(LogOptions? options = null)` → `Stream?` (live; `null` if the log does not exist). Also
   `client.Run(id).GetStreamedLogAsync()`.
 
 Addressed by its own id (`client.Log(buildOrRunId)`) as above, a missing log resolves to `null`. Reached
 through a run or build instead (`client.Run(id).Log()`, `client.Build(id).Log()`), a 404 is ambiguous —
-it could mean the run/build itself or its log is missing — so both methods throw an `ApifyApiException`
-there instead.
+it could mean the run/build itself or its log is missing — so `GetAsync`/`GetBytesAsync` throw an
+`ApifyApiException` there instead.
 
 `LogOptions` fields: `Raw` (`bool?`, return the unprocessed log rather than the parsed form) and
 `Download` (`bool?`, request a download `Content-Disposition`).

docs/storages.md

@@ -31,6 +31,10 @@ reached from a run (`client.Run(id).Dataset()`, `.KeyValueStore()`, `.RequestQue
   item across pages, fetching each page on demand.
 - `DownloadItemsAsync(DownloadItemsFormat format, DatasetDownloadOptions? options = null)` → serialized items as
   `byte[]` (raw bytes, so binary formats like `Xlsx` are not corrupted; decode text formats yourself).
+- `StreamItemsAsync(DownloadItemsFormat format, DatasetDownloadOptions? options = null)` → the same export as a
+  `Stream`, so a large export is processed incrementally instead of buffered into memory. Unlike
+  `DownloadItemsAsync`, this is a single attempt (not retried); dispose the stream when done, which also
+  releases the underlying connection.
 - `PushItemsAsync(object items)` — push one object or an array of objects.
 - `GetStatisticsAsync()` → `JsonObject` (throws if the dataset does not exist).
 - `CreateItemsPublicUrlAsync(DatasetListItemsOptions? options = null, int? expiresInSecs = null, DownloadItemsFormat? format = null)` → `string` (a signed public URL; `format` defaults to `Json`).
@@ -69,6 +73,7 @@ API's export formats:
 
 ```csharp
 using System;
+using System.IO;
 using System.Text;
 using Apify.Client;
 using Apify.Client.Options;
@@ -80,14 +85,21 @@ var page = await client.Dataset(dataset.Id!).ListItemsAsync(new DatasetListItems
 Console.WriteLine(page.Count); // items in this page; page.Total is the count across all pages
 var csvBytes = await client.Dataset(dataset.Id!).DownloadItemsAsync(DownloadItemsFormat.Csv);
 Console.WriteLine(Encoding.UTF8.GetString(csvBytes)); // CSV is text; decode the raw bytes
+
+// Stream a large export to a file instead of buffering it all in memory.
+await using var csvStream = await client.Dataset(dataset.Id!).StreamItemsAsync(DownloadItemsFormat.Csv);
+await using var file = File.Create("items.csv");
+await csvStream.CopyToAsync(file);
 ```
 
 ## Key-value stores
 
 `client.KeyValueStores()` / `client.KeyValueStore(id)`.
 
 - `GetAsync()` → `KeyValueStore?`; `UpdateAsync(object newFields)` → `KeyValueStore`; `DeleteAsync()`.
-- `ListKeysAsync(ListKeysOptions? options = null)` → `KeyValueStoreKeysPage`.
+- `ListKeysAsync(ListKeysOptions? options = null)` → `KeyValueStoreKeysPage` (one page).
+- `IterateKeysAsync(IterateKeysOptions? options = null)` → `IAsyncEnumerable<KeyValueStoreKey>` — lazily
+  iterate every key across pages, fetching each page (sized by `MaxPageLimit`) on demand.
 - `RecordExistsAsync(string key)` → `bool`.
 - `GetRecordAsync(string key, GetRecordOptions? options = null)` → `KeyValueStoreRecord?`.
   `KeyValueStoreRecord.Value` is a `byte[]` of the record's raw bytes (so binary records survive intact);
@@ -101,8 +113,11 @@ Console.WriteLine(Encoding.UTF8.GetString(csvBytes)); // CSV is text; decode the
 
 `ListKeysOptions` fields: `Limit` (page size), `ExclusiveStartKey` (start after this key),
 `Prefix` (only keys with this prefix), `Collection` (a named record collection), and `Signature`
-(for signed public URLs). `GetRecordOptions` fields: `Attachment` (request a download disposition)
-and `Signature`. `SetRecordOptions` fields: `TimeoutSecs` and `DoNotRetryTimeouts`.
+(for signed public URLs). `IterateKeysOptions` fields: `Limit` (total across all pages),
+`MaxPageLimit` (page size; client-side only, never sent to the API — defaults to 1000),
+`ExclusiveStartKey`, `Collection`, and `Prefix`. `GetRecordOptions` fields: `Attachment` (request a
+download disposition) and `Signature`. `SetRecordOptions` fields: `TimeoutSecs`, `DoNotRetryTimeouts`,
+and `ContentEncoding` (upload a value that is already compressed — see below).
 
 ```csharp
 using System;
@@ -128,6 +143,40 @@ if (record is not null)
 await client.KeyValueStore(store.Id!).SetRecordAsync("blob", new byte[] { 0x00, 0xFF }, "application/octet-stream");
 ```
 
+### Uploading a pre-compressed record
+
+`SetRecordOptions.ContentEncoding` uploads a value you already compressed yourself (`gzip`, `br`,
+`deflate`, or `identity`), sent verbatim as the `Content-Encoding` header: the client sends the bytes
+as-is and skips its own compression, so they are never compressed twice.
+
+```csharp
+using System.IO;
+using System.IO.Compression;
+using Apify.Client;
+using Apify.Client.Options;
+
+var client = new ApifyClient("my-api-token");
+var store = await client.KeyValueStores().GetOrCreateAsync("my-store");
+
+byte[] Gzip(byte[] data)
+{
+    using var output = new MemoryStream();
+    using (var gzip = new GZipStream(output, CompressionMode.Compress))
+    {
+        gzip.Write(data, 0, data.Length);
+    }
+
+    return output.ToArray();
+}
+
+var gzippedReport = Gzip(System.Text.Encoding.UTF8.GetBytes("{\"rows\":1000000}"));
+await client.KeyValueStore(store.Id!).SetRecordAsync(
+    "report.json",
+    gzippedReport,
+    "application/json",
+    new SetRecordOptions { ContentEncoding = "gzip" });
+```
+
 ## Request queues
 
 `client.RequestQueues()` / `client.RequestQueue(id, RequestQueueClientOptions? options = null)`. The

src/Apify.Client/Apify.Client.csproj

@@ -6,7 +6,7 @@
 
     <!-- NuGet package metadata (see the publish workflow). -->
     <PackageId>Apify.Client</PackageId>
-    <Version>0.5.0</Version>
+    <Version>0.6.0</Version>
     <Authors>Apify</Authors>
     <Company>Apify</Company>
     <Product>Apify API client for .NET</Product>

src/Apify.Client/ApifyClientVersion.cs

@@ -14,11 +14,11 @@ public static class ApifyClientVersion
     /// The semantic version of this client library (see https://semver.org/). Changes to the public
     /// interface other than additive ones are considered breaking changes.
     /// </summary>
-    public const string ClientVersion = "0.5.0";
+    public const string ClientVersion = "0.6.0";
 
     /// <summary>
     /// The version of the Apify OpenAPI specification this client was generated and verified against.
     /// Corresponds to the <c>info.version</c> field of the Apify OpenAPI document.
     /// </summary>
-    public const string ApiSpecVersion = "v2-2026-10-08T073338Z";
+    public const string ApiSpecVersion = "v2-2026-10-19T133704Z";
 }

src/Apify.Client/Exceptions/ApifyApiException.cs

@@ -13,7 +13,7 @@ namespace Apify.Client.Exceptions;
 /// <see cref="Type"/>, the human-readable <see cref="ApiMessage"/>, the HTTP <see cref="StatusCode"/>,
 /// the number of the final <see cref="Attempt"/>, and the request <see cref="HttpMethod"/>/<see cref="Path"/>.
 /// </remarks>
-public class ApifyApiException : Exception
+public class ApifyApiException : ApifyClientException
 {
     /// <summary>Creates an API exception with the parsed error details.</summary>
     /// <param name="statusCode">The HTTP status code of the error response.</param>

src/Apify.Client/Exceptions/ApifyClientException.cs

@@ -0,0 +1,24 @@
+using System;
+
+namespace Apify.Client.Exceptions;
+
+/// <summary>
+/// Base class of every error this client throws for a failed API call: <see cref="ApifyApiException"/>
+/// (and its per-status subclasses) and <see cref="ApifyTransportException"/>. A single
+/// <c>catch (ApifyClientException)</c> catches all of them, matching the reference client's
+/// <c>ApifyClientError</c>.
+/// </summary>
+/// <remarks>
+/// An invalid argument throws a standard <see cref="ArgumentException"/>/<see cref="ArgumentOutOfRangeException"/>
+/// before any request is sent; neither of those extends this class.
+/// </remarks>
+public abstract class ApifyClientException : Exception
+{
+    /// <summary>Creates a client exception with the given message.</summary>
+    /// <param name="message">A description of the failure.</param>
+    /// <param name="innerException">The underlying exception, if any.</param>
+    protected ApifyClientException(string message, Exception? innerException = null)
+        : base(message, innerException)
+    {
+    }
+}

src/Apify.Client/Exceptions/ApifyTransportException.cs

@@ -9,7 +9,7 @@ namespace Apify.Client.Exceptions;
 /// A custom <see cref="Apify.Client.Http.IHttpTransport"/> should throw this for connection, DNS and
 /// timeout failures; a non-2xx HTTP status must be returned as a normal response instead.
 /// </remarks>
-public sealed class ApifyTransportException : Exception
+public sealed class ApifyTransportException : ApifyClientException
 {
     /// <summary>Creates a transport exception.</summary>
     /// <param name="message">A description of the failure.</param>

src/Apify.Client/Internal/HttpClientCore.cs

@@ -135,6 +135,7 @@ public async Task<HttpResponseMessage> CallAsync(
         byte[]? bodyBytes = null,
         IReadOnlyDictionary<string, string>? extraHeaders = null,
         RequestTier tier = RequestTier.Long,
+        string? explicitContentEncoding = null,
         CancellationToken cancellationToken = default)
     {
         var delayMillis = _retry.MinDelayMillis;
@@ -148,8 +149,12 @@ public async Task<HttpResponseMessage> CallAsync(
         var baseTimeout = timeout ?? TimeSpan.FromSeconds(_retry.TierSecs(tier));
         var overallCap = TimeSpan.FromSeconds(_retry.TimeoutSecs);
         // Normalize (and, when large enough, compress) the body once up front so retries reuse the same
-        // prepared payload instead of re-encoding and re-compressing on every attempt.
-        var prepared = PrepareBody(body, bodyBytes, contentType);
+        // prepared payload instead of re-encoding and re-compressing on every attempt. A caller-supplied
+        // explicitContentEncoding (e.g. SetRecordAsync's contentEncoding for an already-compressed value)
+        // skips this: the bytes are already in their final, encoded form and must reach the API unchanged.
+        var prepared = explicitContentEncoding is not null
+            ? new PreparedBody(bodyBytes ?? (body is not null ? Encoding.UTF8.GetBytes(body) : null), contentType, explicitContentEncoding)
+            : PrepareBody(body, bodyBytes, contentType);
         Exception? lastError = null;
 
         for (var attempt = 1; attempt <= maxAttempts; attempt++)

src/Apify.Client/Internal/ResourceContext.cs

@@ -373,6 +373,36 @@ public async Task<string> GetRawRequiredAsync(string subPath, QueryParams p, Req
         return await response.Content.ReadAsStringAsync(ct).ConfigureAwait(false);
     }
 
+    /// <summary>
+    /// GET returning the raw response body as bytes (no data envelope, no UTF-8 decoding — useful when the
+    /// body is not necessarily text). Returns <c>null</c> on not-found.
+    /// </summary>
+    public async Task<byte[]?> GetRawBytesAsync(string subPath, QueryParams p, RequestTier tier, CancellationToken ct)
+    {
+        var url = MergedParams(p).ApplyToUrl(SubUrl(subPath));
+        try
+        {
+            using var response = await Http.CallAsync(HttpMethod.Get, url, timeout: _requestTimeout, tier: tier, cancellationToken: ct).ConfigureAwait(false);
+            return await response.Content.ReadAsByteArrayAsync(ct).ConfigureAwait(false);
+        }
+        catch (ApifyApiException e) when (HttpClientCore.IsNotFound(e))
+        {
+            return null;
+        }
+    }
+
+    /// <summary>
+    /// GET returning the raw response body as bytes; propagates a 404 instead of swallowing it. The byte
+    /// counterpart of <see cref="GetRawRequiredAsync"/> (see its remarks on when a missing parent should
+    /// surface as an error rather than an ambiguous empty result).
+    /// </summary>
+    public async Task<byte[]> GetRawBytesRequiredAsync(string subPath, QueryParams p, RequestTier tier, CancellationToken ct)
+    {
+        var url = MergedParams(p).ApplyToUrl(SubUrl(subPath));
+        using var response = await Http.CallAsync(HttpMethod.Get, url, timeout: _requestTimeout, tier: tier, cancellationToken: ct).ConfigureAwait(false);
+        return await response.Content.ReadAsByteArrayAsync(ct).ConfigureAwait(false);
+    }
+
     /// <summary>HEAD request; returns whether the resource exists.</summary>
     public async Task<bool> HeadExistsAsync(string subPath, QueryParams p, RequestTier tier, CancellationToken ct)
     {
@@ -388,11 +418,15 @@ public async Task<bool> HeadExistsAsync(string subPath, QueryParams p, RequestTi
         }
     }
 
-    /// <summary>PUT with raw bytes and a content type, with an explicit per-request timeout and retry control.</summary>
-    public async Task PutRawAsync(string subPath, QueryParams p, byte[] body, string contentType, TimeSpan? timeout, bool doNotRetryTimeouts, RequestTier tier, CancellationToken ct)
+    /// <summary>
+    /// PUT with raw bytes and a content type, with an explicit per-request timeout and retry control.
+    /// <paramref name="contentEncoding"/>, when given, is sent verbatim as the <c>Content-Encoding</c>
+    /// header and skips the client's own compression — used for a value that is already compressed.
+    /// </summary>
+    public async Task PutRawAsync(string subPath, QueryParams p, byte[] body, string contentType, TimeSpan? timeout, bool doNotRetryTimeouts, RequestTier tier, CancellationToken ct, string? contentEncoding = null)
     {
         var url = MergedParams(p).ApplyToUrl(SubUrl(subPath));
-        using var response = await Http.CallAsync(HttpMethod.Put, url, null, contentType, timeout ?? _requestTimeout, doNotRetryTimeouts, bodyBytes: body, tier: tier, cancellationToken: ct).ConfigureAwait(false);
+        using var response = await Http.CallAsync(HttpMethod.Put, url, null, contentType, timeout ?? _requestTimeout, doNotRetryTimeouts, bodyBytes: body, tier: tier, explicitContentEncoding: contentEncoding, cancellationToken: ct).ConfigureAwait(false);
     }
 
     // ---- Wait-for-finish ------------------------------------------------------