WP_Query: force deterministic ordering - #10262
Conversation
8f83b7e to
85339ff
Compare
ramonjd
changed the title
| $orderby = ''; | ||
| } else { | ||
| $orderby_array = array(); | ||
| /* |
There was a problem hiding this comment.
@peterwilsoncc When you get a spare moment (can be after 6.9 or whenever you have head space) could you sanity check this approach for me?
The TL;DR is:
When multiple posts have identical values for the primary sort field (like post_date, post_title, menu_order), the database doesn't guarantee consistent ordering across pagination.
This causes inconsistent pagination results, mainly in the form of dupes.
The solution here (and in all the other attempts from 6 years ago) has been to automatically add ID as a secondary sort field when ordering by fields that can have duplicate values. This ensures records with identical primary sort values always appear in the same order.
There was a problem hiding this comment.
You'll also need to consider seeded RAND, see
wordpress-develop/src/wp-includes/class-wp-query.php
Lines 1732 to 1738 in 56868f8
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
Test using WordPress PlaygroundThe changes in this pull request can previewed and tested using a WordPress Playground instance. WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser. Some things to be aware of
For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation. |
9c77315 to
03ff908
Compare
a204b0d to
d7b7c74
Compare
|
I'm going to try to revive this PR with the aim to get it in for 7.0. I'll respond to @peterwilsoncc's feedback first. I'm a bit short on time so if any folks have time to test this approach or help with alternatives, that'd be awesome! |
| sort( $args['post_status'] ); | ||
| } | ||
|
|
||
| // Add a default orderby value of date to ensure same cache key generation. |
There was a problem hiding this comment.
Another nitpick :) This comment still seems okay imho. No need to delete it, just edit it?
|
I was mulling over this today, and given the experience in https://make.wordpress.org/core/2025/12/10/adjacent-post-navigation-changes-in-wordpress-6-9-and-compatibility-issues/, I was wondering if the scope here could be narrowed further to only paginated queries. I'm not sure there's a remedy for filters receiving different values with this PR.
Possibly deterministic ordering could be opt-in to begin with to give folks time to update, or just make it opt-out from the beginning. |
I don't think that's possible as we can't know if a query will be paginated before making the query (eg the blogs home page is likely paginated but doesn't have a page number/offset). There's quite a few plugins with large numbers of installs using those filters, so yeah, let's take a look at these https://wpdirectory.net/search/01KC7TKAN2FGBHSK1NC321HF63 Now there is native caching in WP Core for WP_Query, I think caching plugins should be fine but we'll need to validate it. Some hosts may still be using their own. |
|
Thanks, Peter. Another thought bubble I had was pursuing a fix indirectly. Rather than forcing deterministic ordering, make it an option to sort by ID AND whatever else you want. Wondering if this trac ticket is relevant. Edit:
LOL you can do that anyway. Me dumb. And that wasn't the trac ticket I was looking for. I think there's one flying about. Maybe this one Should REST API support multiple orderby values? |
467d7cf to
ffe31dc
Compare
| * @group ordering | ||
| * @ticket xxxxx | ||
| */ | ||
| class Tests_Query_DeterministicOrdering extends WP_UnitTestCase { |
There was a problem hiding this comment.
I've been trying to write some failing tests proving that when posts have the same date, querying them using per_page and page results in duplicate ids across pages.
I can reproduce on trunk:
Kapture.2025-12-31.at.16.34.17.mp4
And when applying the branch, the bug is no longer there
Kapture.2025-12-31.at.16.36.59.mp4
But I can't create a test scenario that fails on trunk 🤔
I'll keep trying
There was a problem hiding this comment.
Same for REST queries. I can't provide a test scenario in the unit tests because they all pass on trunk.
E.g., creating 20 pages with the same post date and querying them.
BUT manual testing confirms that this branch fixes the bug, for example, given n pages with the same post_date and ordering my menu_order, fetching /wp/v2/pages on trunk
And on this branch
| /* | ||
| * Ensure deterministic ordering to prevent duplicate records across pages. | ||
| * Add ID tie-breaker after filters have been applied, so filters receive | ||
| * the original orderby value (for backward compatibility) and the tie-breaker | ||
| * is preserved even if filters modify the orderby. | ||
| * | ||
| * Note: this is to circumvent a bug that is currently being tracked in | ||
| * https://core.trac.wordpress.org/ticket/44349. | ||
| */ | ||
| if ( ! empty( $orderby ) && $deterministic_orderby_meta['needed'] ) { | ||
| // Check if ID tie-breaker is already present in the orderby string. | ||
| $id_tie_breaker_pattern = '/\b' . preg_quote( $wpdb->posts, '/' ) . '\.ID\b/i'; | ||
| if ( ! preg_match( $id_tie_breaker_pattern, $orderby ) ) { | ||
| // Add ID as tie-breaker at the end. | ||
| $orderby .= ', ' . "{$wpdb->posts}.ID " . $deterministic_orderby_meta['order']; | ||
| } | ||
| } |
There was a problem hiding this comment.
Based on the experience in https://core.trac.wordpress.org/ticket/64390 I've moved the logic beneath the filters
| if ( ! empty( $orderby ) && $deterministic_orderby_meta['needed'] ) { | ||
| /* | ||
| * Only add ID tie-breaker if no filter modified the orderby. | ||
| * If a filter modified it, we assume they know what they're doing and don't interfere. | ||
| */ | ||
| if ( ! empty( $deterministic_orderby_meta['original'] ) && $orderby === $deterministic_orderby_meta['original'] ) { | ||
| // Add ID as tie-breaker at the end. | ||
| $orderby .= ', ' . "{$wpdb->posts}.ID " . $deterministic_orderby_meta['order']; | ||
| } | ||
| } |
There was a problem hiding this comment.
In light of https://core.trac.wordpress.org/ticket/64390 only modify the clause when the filter doesn't.
For backwards compat
…nt post creation. Updated post_date to use str_pad for zero-padding single-digit days.
…post creation. Introduced separate arrays for posts with identical dates, titles, and menu orders to enhance test clarity and maintainability. Updated queries to reference these shared fixtures, ensuring consistent results across pagination and ordering scenarios.
…r deterministic ordering. Updated comments for clarity and introduced a new test for metadata ordering to ensure no duplicates across paginated results.
Introduced new tests to verify deterministic ordering when posts are ordered by search relevance. Created shared fixtures for posts with identical content to ensure consistent relevance scores, preventing duplicates across paginated results. Updated the test suite to include scenarios for both explicit and empty orderby parameters, ensuring robust coverage of search-related ordering behavior.
…Included 'post__in', 'post_name__in', 'post_parent__in', and 'include' to improve query flexibility and support additional ordering scenarios.
… after filters. Updated logic to ensure filters receive the original orderby value, maintaining backward compatibility. Added tests to verify correct behavior when filters modify orderby and prevent duplicate posts across paginated results.
…ssertions. Added a new test to verify the inclusion of ID tie-breaker in the final SQL query for deterministic ordering, ensuring backward compatibility with filters.
…ed related tests to verify that filter modifications are respected.
7019442 to
4896bca
Compare
Posts that share a value for the column being sorted on have no fixed order, so paginated queries can show the same post on two pages and skip another entirely. Appending the ID to ORDER BY breaks those ties. The tie-breaker is now built into the ORDER BY alongside every other clause, before `posts_orderby`, `posts_clauses` and their `_request` counterparts run, matching how WP_Comment_Query has added its own comment_ID tie-breaker since 4.4. Previously it was appended after those filters, and only when the filtered value still matched the string built earlier. That comparison was exact, so a filter returning the same clause with a trailing space silently switched the tie-breaker off. Building it up front removes the comparison, and filters now receive the ORDER BY that actually runs. Ordering that already ends in a unique value gets no tie-breaker: ID, post__in, post_name__in, post_parent__in, rand, and seeded RAND(n). The last of these previously had the ID appended, which defeated the point of passing a seed. Also stop dropping the ORDER BY when a `posts_clauses_request` filter returns no 'orderby' of its own. It now keeps the clause built earlier, including anything `posts_orderby_request` did to it, rather than leaving the query unordered and undoing the tie-breaker. `get_pages()` passes its 'sort_column' as array( 'none' => $sort_order ), so the 'none' check accepts that form as well as the bare string. Props ramonopoly, peterwilsoncc, azaozz. See #44349.
generate_cache_key() rewrote the 'orderby' argument from 'date' to 'date, ID' before hashing. The hash also covers the SQL, which already carries the tie-breaker, so the rewrite changed the key for every default query without distinguishing anything the SQL had not already distinguished. Upgrading would have missed every cached post query for no gain. Restores the comment explaining why the default is set at all. See #44349.
The cache key is built from the query with its SELECT fields normalised to `wp_posts.*`. This split that replacement around the ORDER BY, to stop it also rewriting a field name appearing there. The guard only held for a query containing exactly one ORDER BY; a filter adding a subquery gave three parts and fell through to the same replacement it was meant to avoid. The value is only ever hashed, never run, and $fields is restricted to three known strings just above, so nothing was corrected here. Restores the single replacement, unchanged from before. See #44349.
Rewrites the ordering tests around cases that actually fail without the fix. Two of them page through posts and check that each one is returned exactly once: - Ordering by date, querying two post statuses at once. A single status lets the database read rows straight from the type_status_date index, which ends in ID and so hides the problem; two statuses make it sort the rows itself. This is the case behind most of the reports. - Ordering by menu_order, which no index covers. The rest pin down what the tie-breaker does and does not touch: the direction it takes, ordering that is already unique, seeded and unseeded random, post__in, 'none' from both WP_Query and get_pages(), and search relevance. The filter tests now assert that each clause filter is handed the ORDER BY that will run, that a filtered clause is used exactly as returned, and that a filter changing nothing keeps the tie-breaker, including one that adds only a trailing space. 17 of the 33 fail without the accompanying fix. Props ramonopoly, peterwilsoncc, azaozz. See #44349, #46294.
The cache key is built from the query with its SELECT columns rewritten to `wp_posts.*`, so that two queries differing only in 'fields' share one entry. That rewrite replaced every occurrence, and now that ORDER BY ends in `wp_posts.ID`, a query with 'fields' => 'ids' had its ORDER BY rewritten too. Its key stopped matching the same query asking for full post objects, so the second one missed the cache and ran again. Replaces the first occurrence only, which is the SELECT list. Reverses the removal in [25f35b5]: the greedy replacement was harmless before the ID was added to the default ORDER BY, and is not now. See #44349.
Replaces the x.x.x placeholder on WP_Query::get_posts() and sets the same version on the two methods added alongside it. See #44349.
peterwilsoncc
left a comment
peterwilsoncc
left a comment
There was a problem hiding this comment.
Apparently I left a comment on this back in the day but forgot to actually press submit review.
| $orderby = ''; | ||
| } else { | ||
| $orderby_array = array(); | ||
| /* |
There was a problem hiding this comment.
You'll also need to consider seeded RAND, see
wordpress-develop/src/wp-includes/class-wp-query.php
Lines 1732 to 1738 in 56868f8
Here's me thinking I can YOLO push to this branch and get away with it 😄 Thanks again for keeping an eye on it. TL;DR I'm trying to revive this initiative and review some of my older, hand-written code/assumptions with the power of the latest models, which is a euphemism for saying "me dumb, robot smart". I haven't tested this branch properly either. |
Both places that recognise a random seed now use the same pattern, so a change to one cannot leave the other behind. The previous pattern was anchored and allowed spaces, accepting forms parse_orderby() rejects and rejecting none that it takes, but the two were free to drift apart. Covers the seed forms parse_orderby() honours, and checks that a seeded query returns the same page twice, which appending the ID would break. Follow-up to [3bf4883]. Props peterwilsoncc. See #44349.
Unattached uploads all have a post_parent of 0, so ordering /wp/v2/media by it leaves every attachment tied. Paging through that returned some attachments twice and never returned others. This is the path the media library's DataViews table takes when sorting on "Uploaded to", and the one the block editor reaches through core-data. Fails without the accompanying fix. See #44349, #46294.
The direction is known when each ORDER BY clause is built, so record it there rather than reading it back off the generated SQL with a regular expression. Removes parse_tiebreaker_order(). WP_Query is widely subclassed, so a protected method is close to public API and worth not adding without need. No change in behaviour, including for an array 'orderby' whose clauses sort in different directions, and for post__in and friends, where 'order' is forced empty and the tie-breaker falls back to the default. Follow-up to [3bf4883]. See #44349.
The comments called the appended clause a "tie-breaker", a term core does not otherwise use. WP_Comment_Query solves the same problem and says "to ensure determinate sorting, always include a comment_ID clause", which needs no glossary. Follow that wording, and rename is_unique_orderby() to is_orderby_id() and $found_unique_orderby to $found_orderby_id to match its $found_orderby_comment_id. Two comments also described the code inaccurately. One said the ordering had to "end in" a unique value, when a unique column anywhere in the list is enough: 'orderby' => array( 'ID' => 'DESC', 'title' => 'ASC' ) correctly gets no extra clause. The other said random ordering "cannot produce ties", which is not true of RAND() and is not the reason it is left alone; the reason is that an ID clause would either change nothing or undo a seeded shuffle. Comments and test names only. Follow-up to [3bf4883]. See #44349.
| $orderby = "{$wpdb->posts}.post_date {$query_vars['order']}, {$wpdb->posts}.ID {$query_vars['order']}"; | ||
| } | ||
| } elseif ( 'none' === $query_vars['orderby'] ) { | ||
| } elseif ( 'none' === $query_vars['orderby'] || isset( $query_vars['orderby']['none'] ) ) { |
There was a problem hiding this comment.
'none' === $query_vars['orderby'] only catches the string.
In get_pages() wp_parse_list( 'none' ) gives array( 'none' ), and array_fill_keys() turns that into array( 'none' => 'DESC' )
so the added isset( $query_vars['orderby']['none'] ) catches the array form too.
is_orderby_id() treated 'post_parent__in' and 'post_name__in' as already determinate, but neither is. FIELD( wp_posts.post_parent, ... ) gives every child of the same parent the same sort value, and slugs are only unique within one post type and parent, so both orderings can tie - and paging a set of pages sharing one parent returned a page twice and missed another entirely. Only the ID itself, post__in (a FIELD over the unique ID), and random ordering stay exempt. The appended ID clause now inherits a blank direction instead of turning it into DESC. 'order' is forced empty for the FIELD()-based orderings, whose clauses sort implicitly ascending; the ID should follow them, and today's observed within-group order (ascending ID) is what existing behaviour and the assertions in tests/phpunit/tests/query/results.php already expect. Also corrects the inverted @return description on is_orderby_id(). Follow-up to [3bf4883]. See #44349.
The array check accepted 'none' as ANY key of an array 'orderby', so get_pages( array( 'sort_column' => 'post_title,none' ) ) - which maps to array( 'post_title' => ..., 'none' => ... ) - dropped the whole ORDER BY, where trunk kept the title ordering. An unordered paginated query is the exact failure this branch fixes. 'none' now blanks the clause only as a bare string or as the array's only key, the form get_pages() sends for 'sort_column' => 'none'. Next to real columns it falls through and is skipped like any other unparseable key, so the remaining columns order as requested, ID clause included. Follow-up to [3bf4883]. See #44349.
The rewrite gave the array branch the string branch's post_date fallback, so 'orderby' => array( 'invalid_field' => 'ASC' ) switched from running with no ORDER BY to ordering by post_date and ID. Nothing reported that behaviour and no test pinned the change, so restore the old result: the fallback now applies only to a string 'orderby', where it always has, and an array whose keys all fail to parse produces no ORDER BY. Follow-up to [3bf4883]. See #44349.
Two orderings the fix covers had no test at all: a meta value shared by every post, and /wp/v2/pages paged by menu_order - the scenario from ticket 46294, where pages usually all tie on the default menu_order of 0. Both new tests fail without the fix. The repeat-query test compared the query cache with itself: the second identical query never reached the database. It now runs uncached. The page-walk assertions counted rows, which duplicated posts pad back up to the expected total; counting distinct posts makes a missed post fail the count as well as the set comparison. Also replaces a test docblock that described this branch's own development history as though it were released behaviour, fixes docblock alignment, and uses the US spelling in the cache-key comment. See #44349, #46294.
| * @ticket 44349 | ||
| * @ticket 46294 | ||
| */ | ||
| public function test_get_items_paged_by_menu_order_returns_each_page_once() { |
There was a problem hiding this comment.
This tests at the API layer, and covers trac ticket #46294's report : "rest api fails to paginate page requests correctly when ordering on menu_order"
| $expected_query = explode( ', ', $expected ); | ||
| $expected_query = array_map( | ||
| function ( $item ) use ( $wpdb ) { | ||
| return "{$wpdb->posts}.{$item}"; | ||
| }, | ||
| $expected_query | ||
| ); | ||
|
|
||
| $this->assertStringContainsString( 'ORDER BY ' . implode( ', ', $expected_query ), $this->sql ); |
There was a problem hiding this comment.
the expected string now holds two columns but the old assertion prepended wp_posts. only once, so this splits the expectation and prefixes each column to match the real SQL (wp_posts.post_date DESC, wp_posts.ID DESC).
3 participants
I'm testing out an approach to fix a long-standing issue:
https://core.trac.wordpress.org/ticket/47642
https://core.trac.wordpress.org/ticket/52907
https://core.trac.wordpress.org/ticket/64042
https://core.trac.wordpress.org/ticket/44349
https://core.trac.wordpress.org/ticket/46294
https://core.trac.wordpress.org/ticket/8107 (17 years! 🏅 )
Possibly related too:
https://core.trac.wordpress.org/ticket/46294
https://core.trac.wordpress.org/ticket/52626
This change addresses potential duplicate records across pages when multiple posts share the same value for a field.
It updates WP_Query ordering to ensure deterministic results by adding ID as a secondary sort field.
Reproducing on trunk
Earlier attempts to write a failing test used
orderby=dateon a single post status. That never fails: thetype_status_dateindex ends inID, so the database supplies the tie-breaker by accident. Ordering on a column with no such index does fail.Paging 10 at a time through a media library of 52 unattached uploads, ordered by
parent(all of them sharepost_parent = 0):orderby=parentorderby=menu_order(pages)The
parentcase is what the media library's DataViews table hits when sorting on "Uploaded to", and what the block editor reaches throughcore-data:On trunk page 2 comes back identical to page 1. On this branch the overlap is empty.
Filters
The tie-breaker is built into the ORDER BY before
posts_orderby,posts_clausesand their_requestcounterparts run, matching howWP_Comment_Queryhas added its owncomment_IDtie-breaker since 4.4. Filters receive the ORDER BY that will actually run, and a filtered clause is used exactly as returned.Ordering that is already unique gets no tie-breaker:
ID,post__in,post_name__in,post_parent__in,rand, and seededRAND(n).Tests
17 of the 37 tests in
Tests_Query_DeterministicOrderingfail without the fix, as does the REST media test.