Performant Python pipelines without the Rust
+ + ++ Correct at time of publishing. For up-to-date + performance and feature information, see + rugo.dev. +
++ Called from Python, rugo reads Avro faster than + arrow-avro, Apache Arrow's own Rust reader. + On one thread it won three of four races, + by 1.2× to 1.6×. Left on its defaults, it uses every core + and was 2.3× to 4.5× faster than any Rust + result. +
+ ++ There's a familiar arc in data engineering. The Python + pipeline gets slow, someone benchmarks it, and the + conclusion is always the same: rewrite the hot path in Rust. + Weeks later you have a second language, a build toolchain, + FFI glue, and a team that can't all read the code. +
+ ++ We think that trade is usually unnecessary. To test it, we + raced rugo's Avro reader, called from Python, against the + best Avro readers Rust has, called from Rust. +
+ +The contenders
+ +| Reader | +Called from | +Returns | +
|---|---|---|
| rugo 0.5.4 | +Python | +columnar Morsels | +
| + arrow-avro + 60.0.0 + | +Rust | +Arrow RecordBatches | +
| apache-avro 0.22.0 | +Rust | +one Value per record |
+
+ arrow-avro is the one to beat. It decodes straight into
+ columns, as rugo does, and in our runs it was 2.7× to 16×
+ faster than the older apache-avro crate. We ran it twice:
+ once on its defaults, and once tuned as far as we could push
+ it, with 65,536-row batches and its deflate switched to
+ zlib-rs, a faster Rust inflater than its default.
+
The test
+ +
+ One file: 200,000 rows of a 20-field event record, the mix
+ you'd see in a Kafka topic or a log stream. Longs, doubles,
+ strings, enums, nullable unions, timestamps, dates, a
+ decimal, a fixed(16), a nested record, an array
+ and a map. fastavro wrote it, so no reader is timed on its
+ own output.
+
+ We wrote it twice, once with deflate (21.6 MB) and once + uncompressed (38.2 MB), and read it two ways: +
+ +-
+
- all fields: every one of the 20. +
-
+ 3 fields:
id, +amountandcountry. rugo is + givencolumns; arrow-avro a projection; + apache-avro a reader schema of just those fields. +
+
+ Every reader decoded the same in-memory bytes, so disk wasn't + part of the timing. Each run was a fresh process on one + thread. The readers took turns across five rounds, and the + figures are medians. The machine is an 18-core Apple Silicon + Mac. +
+ +One thread each
+ +Median read time, shorter is faster:
+ +| Race | +rugo (Python) | +arrow-avro | +apache-avro | +rugo vs arrow-avro | +
|---|---|---|---|---|
| deflate, all fields | +85.1 ms | +102.3 ms | +272.0 ms | +1.20× faster | +
| deflate, 3 fields | +52.9 ms | +83.2 ms | +402.0 ms | +1.57× faster | +
| uncompressed, all fields | +47.9 ms | +40.6 ms | +204.3 ms | +1.18× slower | +
| uncompressed, 3 fields | +14.5 ms | +21.0 ms | +335.5 ms | +1.45× faster | +
+ Python won three of the four. The arrow-avro figures are its + tuned build; on its defaults it was 15 ms slower on the + deflate races and level on the uncompressed ones. + apache-avro was never in it: it came last in every race, up + to 23× slower than rugo. +
+ +Every core
+ ++ One thread each keeps the race fair, but it isn't how anyone + runs rugo. On its defaults, rugo spreads a file's blocks + across every core. Neither Rust reader can do that for a + single file, so this compares rugo, untouched, with + arrow-avro at its best: +
+ +| Race | +rugo, defaults | +arrow-avro, tuned | +rugo vs arrow-avro | +
|---|---|---|---|
| deflate, all fields | +30.9 ms | +102.3 ms | +3.3× faster | +
| deflate, 3 fields | +18.3 ms | +83.2 ms | +4.5× faster | +
| uncompressed, all fields | +17.7 ms | +40.6 ms | +2.3× faster | +
| uncompressed, 3 fields | +5.1 ms | +21.0 ms | +4.1× faster | +
That's what you get without changing a setting.
+ +Where we lost
+ ++ One race went to Rust. Reading every field of the + uncompressed file on one thread, arrow-avro took 40.6 ms and + rugo took 47.9 ms, so arrow-avro was 18% faster. +
+ ++ We're showing it because it's the purest test of the decoder + itself: no inflate, no skipping, just bytes into columns. + It's also the one we most want to win. +
+ ++ Our best guess, not yet tested: rugo turns the nested record + and the map into JSON text, which is extra work. arrow-avro + builds native struct and map arrays instead. Reading the file + without those two fields would tell us whether the decoder + itself is behind. +
+ ++ It doesn't change the headline. Kafka Connect, Spark and + Iceberg usually write Avro compressed, and on compressed + files Python wins. +
+ +Why Python can be this fast
+ ++ Python isn't doing the work. rugo is a native C++ engine with + a Python front door: you call it from Python, and the bytes + never pass through the interpreter. +
+ +-
+
- + Compiled schema. rugo turns the Avro + schema into a flat program of decode steps once, then + runs that program over every block. Nothing is looked up + per row. + +
- + Straight into columns. Values land in + typed columnar vectors, with no Python object per value. + That's what you'd write in Rust; you just don't have to. + +
- + Fast inflate. Deflate blocks go through + libdeflate, a dedicated, heavily tuned inflate library. + On compressed files that's most of the time, and most of + the lead. + +
- + Cheap skipping. Fields you don't ask for + are stepped over, not decoded. On the uncompressed file, + reading 3 of 20 fields takes about a third of the time + of reading them all. + +
- + Every core. A file's blocks are + independent, so rugo spreads them across threads and + still hands back batches in file order. + +
+ The point isn't that Rust is slow. arrow-avro is excellent + work. The point is that "fast" is a property of the engine, + not of the language you call it from. +
+ +What this leaves out
+ +-
+
- + One schema. A wide, flat record, or one + that's mostly strings, could move the margins either way. + +
- + One machine. Every figure here is from + Apple Silicon. We haven't run x86 yet. + +
- + Different outputs. rugo returns nested + records and maps as JSON text; arrow-avro returns native + struct and map arrays. Both end in columns, but they + aren't the same columns. + +
- + Other languages. We didn't test Go, + Java or C++ readers. The ones we know of decode a row at + a time, as apache-avro does. + +
- + Reading the file. Every reader got the + bytes already in memory. Reading from disk or object + storage adds the same time to each. + +
Try it
+ +
+ pip install rugo is the whole install. There's
+ no toolchain to set up, and no PyArrow or NumPy:
+
from rugo.avro import read_avro + +with read_avro( + "events.avro", columns=["id", "amount", "country"] +) as reader: + for morsel in reader: + print(morsel.num_rows, morsel.column("amount").to_pylist()[:3])+ +
Reproducing this
+ +
+ Measured 18 October 2026 on an 18-core Apple Silicon Mac,
+ against rugo 0.5.4 from PyPI on CPython 3.14, and arrow-avro
+ 60.0.0 and apache-avro 0.22.0 built with Rust 1.96.1
+ (release profile, fat LTO). The tuned arrow-avro build
+ switches flate2 to zlib-rs
+ 0.6.5; the default build uses miniz_oxide.
+ rugo runs with max_threads=1 for the
+ one-thread races.
+
+ Each of the five rounds runs every reader in a fresh + process, five reads each, with the order rotated each round + so no reader always goes first. The figures are the median + of the per-process medians. +
+ +
+ The harness, the data generator and the Rust readers are in
+ benchmarks: measure_avro_rust.py builds both Rust
+ readers, writes the file and runs the race on your machine.
+ We'd like to hear what you get, especially on x86.
+