Higher memory footprint after switching from Goliath/EventMachine to Falcon/Async

[tleish]

We’re evaluating Falcon as a replacement for Goliath/EventMachine and ran into something we weren’t expecting. (We tried switching a while ago, but ran into some issues with a customer).

We temporarily migrated our API servers from Goliath/EventMachine to Falcon/Async. CPU utilization stayed roughly the same (if anything, slightly lower), but memory usage increased significantly and remained elevated until we switched back.

During the migration:

• We gradually moved traffic from Goliath to Falcon.
• As more servers were moved to Falcon, memory usage increased across those hosts.
• CPU stayed within the same general range as before.
• Once we rolled all servers back to Goliath, memory immediately returned to the previous baseline.

We’re trying to understand whether this is expected behavior or if there’s something we’re missing.

A few questions:

• Are there any common causes of higher memory usage when moving from EventMachine to Falcon/Async?
• Is there anything specific we should profile?
• Are there Falcon or Async configuration options that commonly affect memory footprint?
• Are there known patterns or pitfalls that can lead to memory growth under sustained load?
• We are running on slightly older gem versions (see below), so maybe there was a known issue with an other version?

If there are any recommended profiling tools or diagnostics you’d like us to collect, we’re happy to do that.

Thanks!

• falcon 0.54.3
• async 2.37.0
• ruby 3.2.10

Memory

CPU

[ioquatix]

Are you using the preload option and the bundle preload group? https://socketry.github.io/falcon/guides/performance-tuning/index#preloading

You want to make sure as much immutable stuff is pre-loaded, e.g. gems, application code, etc, e.g.

# config/preload.rb

require_relative "config/application"

# gems.rb / Gemfile

group :preload do
  gem "rails"
  # ... pretty much every production gem should be here unless they are unsafe to load.
end

Then, in your service definition:

service "my-app" do
  preload "config/preload.rb"
end

Note that preloads are not isolated per service, as services are all launched from the same root controller (process).

Assuming you've done that, you might be leaking resources.

Make sure you don't leak thread locals, requests or clients, ensure you are closing resources. Probably the next step if you have leaks is to evaluate what may be leaking by using a memory profiler. Let me know if you need advice about how to do this.

[tleish]

Maybe I'm misunderstanding this, it seems preload's benefit is entirely a function of instance count. If that's the case, with our current deployment model, both Goliath and Falcon are only running 1 instance per server. The memory from the graphs above are a memory consumption per server.

[samuel-williams-shopify]

You're right: if Falcon is explicitly running with a worker count of one, preloading won't materially explain the difference. I may have conflated one deployed Falcon service with one application worker.

Falcon 0.54.3 defaults to one worker per processor, so could you share the launch command or falcon.rb configuration, along with the process tree? A single systemd service or container may still contain a controller and several application workers.

Before investigating the older stack in detail, I recommend migrating to the latest Falcon release (currently 0.55.6) and updating the related Socketry dependencies. Falcon 0.55 replaces async-container-supervisor with async-service-supervisor; the migration is described in the async-service-supervisor migration guide. This may not resolve the memory behaviour by itself, but it gives us a current, supported baseline and access to the diagnostics below.

The graph looks like it may show two separate effects: an immediate increase in baseline memory as each host moves to Falcon, followed by continued growth under load. We should distinguish additional process overhead from retained objects before concluding that the latter is a leak.

The ProcessMonitor from process-metrics is a good first step. It captures the entire process tree and, where supported by the operating system, reports both resident_size and proportional_size. The proportional size (PSS) is particularly useful for Falcon's forked workers because summing RSS across processes counts shared pages more than once.

Could you enable it at a suitable interval and collect the per-process metrics during another test?

service "supervisor" do
	include Async::Service::Supervisor::Environment

	monitors do
		[
			Async::Service::Supervisor::ProcessMonitor.new(interval: 60)
		]
	end
end

It would be useful to see:

• The configured worker count and resulting process tree.
• PSS and RSS for the controller and each worker over time.
• Whether individual worker PSS continues growing or eventually plateaus.
• Approximate request concurrency and whether there are long-lived or streaming connections.

That should tell us whether this is fixed overhead from multiple workers, growth isolated to particular workers, or memory outside Ruby's managed heap.

If an individual worker's PSS continues growing, you can use the worker-level diagnostics in async-service-supervisor to list the live workers and capture comparable heap snapshots from the same worker:

$ bake async:service:supervisor:workers

$ bake async:service:supervisor:memory_dump \
    connection_id=1 \
    path=/var/tmp/worker-1-before.json

# Run representative traffic or wait through the suspected growth period.

$ bake async:service:supervisor:memory_dump \
    connection_id=1 \
    path=/var/tmp/worker-1-after.json

A two- or three-snapshot sheap comparison can identify objects retained between snapshots. For a bounded capture that includes allocation locations:

$ bake async:service:supervisor:allocation_trace_start connection_id=1

# Exercise the workload.

$ bake async:service:supervisor:allocation_trace_stop \
    connection_id=1 \
    path=/var/tmp/worker-1-traced.json

The Memory Diagnostics guide covers the complete workflow, including heap-profiler, sheap, and reap. Heap dumps can contain application data, so please don't attach the raw files publicly; summaries of the dominant classes and retention paths would be sufficient.

If the Ruby heap remains stable while PSS continues growing, that would instead point us toward native buffers, allocator fragmentation, or another source outside Ruby's object heap.

[tleish]

Wow... thank you for all the suggestions. I'll need some time to work through this and test in production, and then I'll report findings here.