Separate named uploads from custom fields

CHANGELOG.md

@@ -1,3 +1,10 @@
+### Unreleased
+
+- Add `uploads: {name => io}` to Assembly options and `create!` for named multipart uploads, separate from signed JSON Instructions.
+- Send custom `fields` only in signed JSON, preserving numbers, booleans, arrays, objects, and null without duplicate form fields.
+- Keep IOs in legacy `fields` working as named uploads throughout v4, omit them from JSON, and warn to migrate to `uploads` before the next major release. Positional `file_N` uploads and `create!(fields: metadata)` overrides remain supported.
+- Restore in-memory uploads to their original position on rate-limit retries and raise a clear `IOError` when a non-seekable upload cannot be retried.
+
 ### 4.0.0 / 2026-09-03
 
 - BREAKING: Replace the long-dormant `rest-client` transport with Faraday 2 and `faraday-multipart`.

Gemfile

@@ -5,4 +5,8 @@ gem "minitest"
 gem "simplecov"
 gem "simplecov-cobertura"
 
+# Ruby 3.1 resolves MultiJson 1.19, which cannot parse JSON 3 (ruby-sdk#89).
+# Keep its test bundle compatible without restricting consumers on newer Rubies.
+gem "json", "< 3.0" if RUBY_VERSION.start_with?("3.1.")
+
 gemspec

README.md

@@ -171,6 +171,38 @@ files = [open('puppies.jpg'), open('kittens.jpg'), open('ferrets.jpg')]
 response = assembly.create! *files
 ```
 
+For names that match a Step's `use.fields` selector, pass `uploads: {name => io}`
+to `assembly` or `create!`. Uploads are multipart parts and are excluded from
+the signed JSON Instructions. Open files in binary mode and close them after
+the request; the SDK leaves caller-owned files open.
+
+```ruby
+File.open('puppies.jpg', 'rb') do |photo|
+  response = transloadit.assembly(
+    steps: {
+      resize: {
+        robot: '/image/resize',
+        use: {fields: ['photo']},
+        width: 320,
+        height: 240
+      }
+    },
+    uploads: {photo: photo},
+    fields: {count: 2, enabled: false, details: {caption: 'Puppies'}}
+  ).create!
+end
+```
+
+Alternatively, use `assembly.create!(uploads: {photo: photo}, fields: metadata)`.
+Per-call `uploads` and `fields` each replace the corresponding Assembly option;
+`uploads: {}` clears only the `uploads` option. Legacy IOs configured in `fields`
+remain uploads for compatibility. Positional files still use `file_0`,
+`file_1`, and so on, and take precedence over named uploads with those names.
+Upload values must respond to `read`; the names `params` and `signature` are reserved.
+For rate-limit retries, in-memory uploads must also support `pos` and `seek`;
+the SDK restores their original position. A non-seekable upload raises `IOError`
+if a retry is needed, instead of sending consumed bytes again.
+
 ### 3. Parallel Assembly
 
 Transloadit allows you to perform several processing steps in parallel. You
@@ -226,9 +258,10 @@ has some nice examples for that.
 
 ### 5. Using fields
 
-Transloadit allows you to submit form field values that you'll get back in the
-notification. This is quite handy if you want to add additional custom metadata
-to the upload itself. You can use fields like the following:
+Transloadit allows you to submit custom metadata that you'll get back in the
+notification. `fields` is included once in the signed JSON Instructions and is
+never duplicated as form fields. Numbers, booleans, arrays, objects, and null
+retain their JSON types. You can use fields like the following:
 
 ```ruby
 require 'transloadit'
@@ -246,6 +279,17 @@ transloadit.assembly(
 ).create! open('/PATH/TO/FILE.mpg')
 ```
 
+`create!(fields: metadata)` also sets custom fields and replaces any metadata
+configured on the Assembly. Uploads and metadata may use the same name.
+
+For compatibility, IO values passed directly in a `fields` hash still upload
+under their field names throughout v4, but emit a deprecation warning. They are
+excluded from JSON; other values in that hash remain custom metadata. Migrate
+from `assembly(fields: {photo: file, count: 2})` to
+`assembly(uploads: {photo: file}, fields: {count: 2})` before the next major release.
+Explicit `uploads` take precedence over legacy IOs with the same name. Existing
+`assembly(fields: {photo: file}).create!(fields: metadata)` calls continue working.
+
 ### 6. Notify URL
 
 If you want to be notified when the processing is finished you can provide
@@ -464,6 +508,11 @@ for the latest [git main](https://rubydoc.info/github/transloadit/ruby-sdk/main/
 Please see [ci.yml](https://github.com/transloadit/ruby-sdk/tree/main/.github/workflows/ci.yml) for a list of supported ruby versions. It may also work on older Rubies, but support for those is not guaranteed. If it doesn't work on one of the officially supported Rubies, please file a
 [bug report](https://github.com/transloadit/ruby-sdk/issues). Compatibility patches for other Rubies are welcome.
 
+On Ruby 3.1, add `gem "json", "< 3.0"` to your application's Gemfile while
+[the older MultiJson/JSON compatibility issue](https://github.com/transloadit/ruby-sdk/issues/89)
+is open. Ruby 3.2 and newer can use JSON 3 with
+[MultiJson 1.21.2 or newer](https://github.com/sferik/multi_json/releases/tag/v1.21.2).
+
 ### Ruby 2.x
 
 If you still need support for Ruby 2.x, 2.0.1 is the last version that supports it.

lib/transloadit.rb

@@ -79,6 +79,10 @@ def step(name, robot, options = {})
   # @option options [String] :template_id the ID of a
   #   {template}[https://transloadit.com/templates] to use instead of
   #   specifying options here directly
+  # @option options [Hash] :fields custom metadata, retained as typed JSON;
+  #   passing upload IOs here is deprecated in favor of :uploads
+  # @option options [Hash<String, IO>] :uploads named multipart uploads, kept
+  #   separate from the signed JSON Instructions
   #
   def assembly(options = {})
     Transloadit::Assembly.new(self, options)

lib/transloadit/assembly.rb

@@ -44,25 +44,45 @@ def steps
   #   @option params [String] :template_id the ID of a
   #     {template}[https://transloadit.com/templates] to use instead of
   #     specifying params here directly
+  #   @option params [Hash] :fields custom metadata, retained as typed JSON
+  #   @option params [Hash<String, IO>] :uploads named multipart uploads;
+  #     overrides the Assembly's uploads when supplied (use {} to clear them)
   #
   def create!(*ios, **params)
     params[:steps] = _wrap_steps_in_hash(params[:steps]) unless params[:steps].nil?
 
+    legacy_uploads, = _split_fields(options[:fields])
+    if params.key?(:fields)
+      uploads, params[:fields] = _split_fields(params[:fields])
+      legacy_uploads.merge!(uploads)
+    end
+    unless legacy_uploads.empty?
+      warn "Passing upload IOs in fields is deprecated; use uploads: {name => io} instead."
+    end
+
     extra_params = {}
-    extra_params.merge!(options[:fields]) if options[:fields]
+    [legacy_uploads, params.delete(:uploads) { options[:uploads] }].compact.each do |uploads|
+      uploads.each do |name, io|
+        name = name.to_s
+        raise ArgumentError, "Upload name #{name.inspect} is reserved" if %w[params signature].include?(name)
+        raise ArgumentError, "Upload #{name.inspect} must be an IO responding to read" unless io.respond_to?(:read)
+
+        extra_params[name] = io
+      end
+    end
+    # Keep positional file_N uploads and their precedence for existing callers.
+    ios.each_with_index { |io, i| extra_params["file_#{i}"] = io }
+    upload_positions = _upload_positions(extra_params.values)
 
     trials = options[:tries] || DEFAULT_TRIES
     (1..trials).each do |trial|
-      # update the payload with file entries
-      ios.each_with_index { |f, i| extra_params.update "file_#{i}": f }
-
       response = _do_request(
         "/assemblies", params, "post", extra_params
       ).extend!(Transloadit::Response::Assembly)
 
       return response unless response.rate_limit?
 
-      _handle_rate_limit!(response, ios, trial < trials)
+      _handle_rate_limit!(response, upload_positions, trial < trials)
     end
   end
 
@@ -123,14 +143,37 @@ def replay_notification(id, params = {})
   # @return [Hash] a Transloadit-compatible Hash of the Assembly's contents
   #
   def to_hash
-    options.merge(
+    result = options.merge(
       auth: transloadit.to_hash,
       steps: steps
-    ).delete_if { |k, v| v.nil? }
+    )
+    result.delete(:uploads)
+    _, result[:fields] = _split_fields(result[:fields]) if result[:fields]
+    result.delete_if { |_, value| value.nil? }
   end
 
   private
 
+  # Legacy fields may contain both named IOs and metadata. Only metadata belongs
+  # in the signed JSON; the IOs remain supported as uploads throughout v4.
+  def _split_fields(fields)
+    return [{}, nil] if fields.nil?
+
+    fields.partition { |_, value| value.respond_to?(:read) }.map(&:to_h)
+  end
+
+  def _upload_positions(uploads)
+    uploads.filter_map do |io|
+      # The transport reopens path-backed uploads for each attempt.
+      next if io.respond_to?(:path) && io.path
+
+      position = io.pos if io.respond_to?(:pos) && io.respond_to?(:seek)
+      [io, position]
+    rescue IOError, SystemCallError
+      [io, nil]
+    end
+  end
+
   #
   # Returns a Transloadit-compatible Hash wrapping the +steps+ passed to it.
   # Accepts any supported format the +steps+ could come in.
@@ -163,14 +206,21 @@ def _wrap_steps_in_hash(steps)
   end
 
   #
-  # Stays idle for certain time and then reopens assembly files for reprocessing.
+  # Restores in-memory upload positions and waits before retrying.
   # Should be called when assembly rate limit is reached.
   #
   # @param  [Response] response  assembly response that comes with a rate limit
-  # @param [Array<IO>] ios the files sent for the assembly to process.
+  # @param [Array<Array>] upload_positions IOs and their positions before upload
   #
-  def _handle_rate_limit!(response, ios, is_retrying)
+  def _handle_rate_limit!(response, upload_positions, is_retrying)
     if is_retrying
+      upload_positions.each do |io, position|
+        raise IOError if position.nil?
+
+        io.seek(position)
+      rescue IOError, SystemCallError
+        raise IOError, "Cannot retry a non-seekable upload; use a file or an IO supporting pos and seek"
+      end
       warn "Rate limit reached. Waiting for #{response.wait_time} seconds before retrying."
       sleep response.wait_time
     else

test/unit/transloadit/test_assembly.rb

@@ -121,7 +121,7 @@
 
         assert_requested(:post, "https://api2.transloadit.com/assemblies") do |req|
           values = values_from_post_body(req.body)
-          _(values["tag"]).must_equal "ninja-cat"
+          _(values.keys).must_equal ["params"]
           _(MultiJson.load(values["params"])["fields"]["tag"]).must_equal "ninja-cat"
         end
       end