Skip to content

array_push(): Append to packed arrays in bulk - #24157

Open
mehmetcansahin wants to merge 5 commits into
php:masterfrom
mehmetcansahin:array-push-packed-fill
Open

mehmetcansahin wants to merge 5 commits into
php:masterfrom
mehmetcansahin:array-push-packed-fill

Conversation

@mehmetcansahin

@mehmetcansahin mehmetcansahin commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

array_push() used to call zend_hash_next_index_insert() for each value. With two or more values it now calls the new zend_hash_next_index_insert_copies(), which inserts all of them and adds a reference to each. When the array is packed or uninitialized and its next free index is right after the last used slot, that function makes room once if needed and stores the values directly. This mainly helps array_push($a, ...$values).

A single value still goes through zend_hash_next_index_insert(): passing it through the new function made array_push($list, $x) about 5% slower.

zend_hash_next_index_insert_copies() inserts the values one by one for: hash arrays; packed arrays whose next free index is past the last used slot (e.g. after unset() of the last element); packed arrays with holes that would need to grow (growth may turn them into a hash); more values than fit below HT_MAX_SIZE.

Keys, the next free index, the internal pointer, table growth and the error for an occupied next index stay the same.

Benchmark: script. Release build (-O2 -DNDEBUG, NTS), OPcache/JIT off (php -n), 15 alternating base/PR pairs per case (best of 5 repetitions per process), base 6fc4017a940 (the master merged into this branch). Apple M1 (macOS, Apple clang) measured PR f1c7c2542fd; AMD EPYC 9V74 (Linux VM, GCC 14.2, pinned to one CPU) measured 67c59787d96, which only changes a comment: its zend_hash.o on M1 and its PHP binary on EPYC are byte-identical to those of f1c7c2542fd.

Times are medians in ns/op. Change = PR median / base median - 1, calculated before rounding. Negative means faster.

Scenario M1 base ns/op M1 PR ns/op Change EPYC base ns/op EPYC PR ns/op Change
packed, 1 value 12.55 12.66 +0.9% 16.03 16.37 +2.1%
packed, 3 values 21.62 17.97 -16.9% 26.01 22.68 -12.8%
packed, 16 values via spread 124.38 83.23 -33.1% 114.59 85.85 -25.1%
empty, 1000 values via spread 6750.33 3947.92 -41.5% 6359.96 3944.01 -38.0%
packed, 1 string 14.98 14.95 -0.2% 17.53 17.57 +0.2%
empty, 1 value 28.44 28.37 -0.2% 30.70 31.81 +3.6%
empty, 2 values 35.41 31.98 -9.7% 38.31 36.47 -4.8%
packed with holes, 1 value 12.57 12.69 +0.9% 16.14 16.27 +0.8%
hash, 1 value 15.14 15.17 +0.2% 18.67 18.01 -3.5%
hash, 3 values 31.35 30.06 -4.1% 36.84 34.12 -7.4%

Base-against-base runs before and after these pairs stayed within ±0.32% on M1 for every case, and ranged from -5.3% to +4.1% on EPYC, where 750 of 900 processes saw steal time above 5%. On EPYC the rows with 3 or more values on a list or [] are well outside that range; the single-value rows are within it, so they don't show a change either way.

On M1 the two +0.9% rows (0.11 ns/call) are outside its range, but on the single-value path array_push() executes no more instructions than on master, and zend_hash_next_index_insert() is unchanged, just placed 48 bytes later in the binary, so I attribute them to code layout.

Hash arrays with several values also gain a little (-4.1% / -7.4%), most likely because the one-by-one path inside the new function inlines the insert instead of calling zend_hash_next_index_insert() for each value.

The bench-common.php verify outputs match for base and PR on both machines.

When the next index of a packed array is the end of its used slots,
reserve the room once and copy the values instead of inserting them one
by one. Arrays with holes only take this path when no growth is needed.

A single value pushed onto an uninitialized array still goes through
zend_hash_next_index_insert(), which initializes the array inline and is
cheaper than a separate initialization.
@mehmetcansahin
mehmetcansahin marked this pull request as ready for review October 6, 2026 09:22
@mehmetcansahin
mehmetcansahin requested a review from bukka as a code owner October 6, 2026 09:22

@LamentXU123 LamentXU123 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just my 2 cents.

Comment thread ext/standard/array.c Outdated
Comment thread ext/standard/array.c Outdated
Comment thread ext/standard/array.c Outdated
Drop UNEXPECTED() on the uninitialized array branch, pushing several
values onto an empty array is common. Fix the stale comment on the
insertion loop and trim the comments that restate the code.

@LamentXU123 LamentXU123 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM /cc @arnaud-lb Thanks!

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

AWS x86_64 (c6id.metal)

Attribute Value
Environment aws
Instance type c6id.metal
Architecture x86_64
CPU Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz, 64 cores @ 2900 MHz
CPU settings disabled deeper C-states, disabled turbo boost, disabled hyper-threading
RAM 251 GB
Kernel 6.18.38-76.139.amzn2023.x86_64
OS Amazon Linux 2023.12.20260727
GCC 14.2.1
Binary layout strategy bolt-align
Time 2026-10-07 10:46:02 UTC
Job details https://github.com/php/php-src/actions/runs/37609520445 (Artifacts)
Changeset https://github.com/php/php-src/compare/421347dff4..9d1523cb6e

Laravel 12.11.0 demo app - 50 iterations, 50 warmups, 100 requests (sec)

PHP Min Max Std dev Rel std dev % Mean Mean diff % Median Median diff % Skewness Z-stat P-value Memory
PHP - baseline@421347d 0.37994 0.38091 0.00019 0.05% 0.38022 0.00% 0.38016 0.00% 1.523 0.000 1.000 31.52 MB
PHP - array-push-packed-fill 0.37984 0.38128 0.00024 0.06% 0.38017 -0.01% 0.38010 -0.01% 2.298 2.099 0.036 31.52 MB

Symfony 2.8.0 demo app - 50 iterations, 50 warmups, 100 requests (sec)

PHP Min Max Std dev Rel std dev % Mean Mean diff % Median Median diff % Skewness Z-stat P-value Memory
PHP - baseline@421347d 0.68935 0.70103 0.00169 0.24% 0.69017 0.00% 0.68982 0.00% 5.811 0.000 1.000 31.53 MB
PHP - array-push-packed-fill 0.68932 0.69246 0.00048 0.07% 0.68987 -0.04% 0.68978 -0.01% 3.482 0.486 0.627 31.52 MB

Wordpress 6.9 main page - 50 iterations, 20 warmups, 20 requests (sec)

PHP Min Max Std dev Rel std dev % Mean Mean diff % Median Median diff % Skewness Z-stat P-value Memory
PHP - baseline@421347d 0.59425 0.59546 0.00033 0.05% 0.59471 0.00% 0.59463 0.00% 0.690 0.000 1.000 31.55 MB
PHP - array-push-packed-fill 0.59442 0.59552 0.00028 0.05% 0.59499 0.05% 0.59500 0.06% 0.036 -4.205 0.000 31.53 MB

bench.php - 50 iterations, 20 warmups, 2 requests (sec)

PHP Min Max Std dev Rel std dev % Mean Mean diff % Median Median diff % Skewness Z-stat P-value Memory
PHP - baseline@421347d 0.45186 0.45651 0.00064 0.14% 0.45277 0.00% 0.45273 0.00% 4.044 0.000 1.000 31.55 MB
PHP - array-push-packed-fill 0.45291 0.45467 0.00040 0.09% 0.45371 0.21% 0.45370 0.21% 0.071 -7.842 0.000 31.53 MB

@ArtUkrainskiy

Copy link
Copy Markdown
Contributor

I measured this on master c65321c (d8ef5ef merges cleanly) with GCC 13.3, GCC 14.2, clang 18.1 and clang 20.1 on an i7-13700H, built with the flags of the official Docker image.

Change in CPU cycles per call with the PR, relative to master (negative = fewer cycles, faster); median of five alternating runs, each call in a loop, so the percentages are of the whole statement:

call master, CPU cycles per call (GCC 13.3) GCC 13.3 GCC 14.2 clang 18.1 clang 20.1
array_push($a, $i), one int; $a starts again from [] every 1,024 calls 60 −2% −2% 0% −7%
array_push($a, $s), one string; same 62 −5% −2% 0% −6%
array_push($a, $i, $i, $i); same 94 −14% −13% −9% −16%
array_push($a, ...$v), 16 ints; same 538 −25% −25% −26% −31%
array_push($a, $i), list with a hole (unset($a[3])) 61 −3% −2% −1% −6%
array_push($a, $i) onto ['k' => 0] (hash, old code path) 78 0% +1% −3% 0%
array_push($a, $i, $i, $i) onto ['k' => 0] 130 −1% +1% +3% +1%
$a = []; array_push($a, $i); 91 +3% +2% −2% +1%
$a = []; array_push($a, $i, $i); 111 −5% −8% −10% −11%
$a = []; array_push($a, ...$v);, 16 ints 575 −23% −19% −25% −31%
$a = []; array_push($a, ...$v);, 1,000 ints 23,735 −36% −36% −35% −38%

Pushing three or more values onto a list or [] takes 9–38% fewer cycles (19–37% fewer instructions). One value onto a list is 0–7% faster, so the common call does not get slower on any of the four compilers. The paths that keep the old loop (a hash array, one value onto []) are at −3 … +3%, from 5–14 more instructions per call for the new checks; master against master in the same series gives −2 … +1%.

The spread rows gain less here than in the PR description: 25–31% fewer cycles for 16 values onto a list and 35–38% for 1,000 values onto [], against 32% and 41–47% less time there (M1 and Xeon). I did not look into why; the base commit, the metric (cycles against ns) and the machines differ.

empty, 2 values: here it takes 5–11% fewer cycles with all four compilers (6–8% fewer instructions), so on this machine the separate zend_hash_real_init_packed() call does not cancel out the saved insert.

Code: the fast path repeats zend_hash internals in array_push() (the ZEND_LONG_MIN next-index rule, growing the packed table, restoring the internal pointer that ZEND_HASH_FILL_END() resets). Moving it into a function, e.g. a zend_hash.c API for appending n values to a packed array, would keep that logic in one place, so I measured that shape: the same code in a static zend_never_inline helper, with the flag check left in array_push(). One int or string onto a list then takes 5% more cycles than master with GCC 14.2 and 7% more with clang 18.1 (one value onto a list with a hole: 4% and 15%), against −7 … +1% with the PR in that series, and the paths that keep the old loop are within one point of the PR. So the inline version in the PR is the faster one.

Correctness: on 13,879 array_push() calls I recorded the return value, keys, values, key()/current(), the next free index and memory_get_usage() after each one, and the output is the same as master with all four compilers and under ASan/UBSan (there apart from memory_get_usage(), which a debug build changes). The calls cover 57 starting arrays (empty, negative next index, unset last/middle/first, after array_pop()/array_shift(), internal pointer in the middle and past the end, holes with and without room, PHP_INT_MAX next index, lists of 1–1,025 elements) with 0–1,025 values each, copy-on-write, self-push, references, foreach by reference while pushing, and 3,000 random sequences of push/unset/pop/shift/next()/end()/copy. The harness catches seven of seven deliberate mutations. ext/standard/tests/array under ASan pass.

Tests: three of those mutations pass ext/standard/tests/array with the new test, even under ASan: the room check one too large (argc > room + 1), the table one slot short (zend_hash_extend(ht, ht->nNumUsed + argc - 1, true)), and removing HT_IS_WITHOUT_HOLES(ht). Two short tests catch all three:

// needs one slot more than the table has: ASan catches both off-by-one mutations
$a = range(1, 7);
var_dump(array_push($a, 8, 9), $a === range(1, 9));   // int(9) bool(true)
$a = range(1, 8);
var_dump(array_push($a, ...range(9, 17)), $a === range(1, 17)); // int(17) bool(true)

// a full packed array with holes still takes the old loop: the internal pointer stays past the end
$a = range(0, 15);
for ($i = 0; $i < 14; $i++) unset($a[$i]);
end($a);
next($a);
var_dump(array_push($a, 16, 17), key($a)); // int(4) NULL; without the holes check: int(4) int(16)

No correctness issues found. LGTM.

Full tables, scripts and data: https://github.com/ArtUkrainskiy/php-src-bench/tree/main/reports/pr-24157-array-push

Comment thread ext/standard/tests/array/array_push_packed_bulk_metadata.phpt Outdated
$a[] = 2;
var_dump(array_keys($a));

// A negative next index of an empty copy is used

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't reach new code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it takes the one-by-one path: the copy is uninitialized with nNextFreeElement = -2. I kept it on purpose, it fails if the next-index check is dropped.

Comment thread ext/standard/array.c Outdated
/* Pushing nothing must leave the array untouched. A single value is added
* to an uninitialized array more cheaply by zend_hash_next_index_insert(). */
if (argc > 0 && (HT_IS_PACKED(ht) || (argc > 1 && (HT_FLAGS(ht) & HASH_FLAG_UNINITIALIZED)))) {
zend_long next_index = ht->nNextFreeElement == ZEND_LONG_MIN ? 0 : ht->nNextFreeElement;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This leaks details of zend_hash into ext/standard, this should be abstracted somehow

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it into Zend/zend_hash.c as a new ZEND_API function, zend_hash_next_index_insert_copies() (noted in UPGRADING.INTERNALS), next to _zend_hash_index_add_or_update_i(), whose next-index and growth rules it follows. array_push() has no HashTable internals left. Since the function writes the slots itself instead of using ZEND_HASH_FILL_*, the internal pointer no longer needs saving and restoring.

It also adds the references, like ZVAL_COPY(): array_push() needs that, and a separate addref loop made 1000 spread values ~6% slower. SplFixedArray's get_properties_for() and __serialize() fill arrays the same way; switching them locally made (array) $fixed 53% and __serialize() 70% faster for 1000 elements, which I'd send as a follow-up.

The call is out of line now, which costs a few points for 3 values compared with the inline version; single values still use zend_hash_next_index_insert() directly. Happy to change the name or semantics if you'd prefer something else.

Comment thread ext/standard/array.c Outdated

/* Pushing nothing must leave the array untouched. A single value is added
* to an uninitialized array more cheaply by zend_hash_next_index_insert(). */
if (argc > 0 && (HT_IS_PACKED(ht) || (argc > 1 && (HT_FLAGS(ht) & HASH_FLAG_UNINITIALIZED)))) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition is quite complex. Maybe the single value is indeed an edge case that's handled faster by the old code, but does it really have such a large performance difference that this complexity is warranted?

@mehmetcansahin mehmetcansahin Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped the argc > 1 special case, it made no measurable difference. The one split left is argc == 1 in array_push(): routing a single value through the new function cost ~5% on array_push($list, $x).

Comment thread ext/standard/array.c Outdated
Add zend_hash_next_index_insert_copies(), which inserts several values
like repeated zend_hash_next_index_insert() calls and adds a reference to
each. Values appended right after the last used slot of a packed or
uninitialized array are stored in one go. array_push() uses it for two or
more values and no longer touches HashTable internals; a single value
still goes through zend_hash_next_index_insert().

Rewrite the test to cover the room and growth boundaries, holes, the next
index, refcounts and a partial insert.
@mehmetcansahin

mehmetcansahin commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Updated in f1c7c25: the bulk insertion now lives in Zend/zend_hash.c (see the review thread). The test covers the room and growth boundaries, holes and the next index, including the mutations @ArtUkrainskiy listed; a table near HT_MAX_SIZE would need ~16 GiB, so that one isn't in the phpt. New M1 numbers are in the description; x86-64 (AMD EPYC) numbers will follow.

@ndossche could you take another look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants