Fix hourly time labels dropping an hour on DST transition days

[jaideeppyne]

Changes

On a day that a site's timezone moves its clocks forward, the hourly graph silently loses the last hour of the day.

time_labels/1 built the time:hour labels by adding 0..n hours to the naive start of the range, where n came from DateTime.diff(last, first, :hour). That diff is absolute, so a 23 hour day yields 23 labels ending at 22:00. ClickHouse buckets the rows with toStartOfHour(toTimeZone(timestamp, tz)), which does produce a 23:00 bucket, so that row has no label to land on. The same range also emits a label for 02:00, an hour that never happened. Fall-back days get the opposite: 25 labels for 24 buckets, the last one belonging to the next day.

New York, 2024-03-10, before:

["2024-03-10 00:00:00", "2024-03-10 01:00:00", "2024-03-10 02:00:00", ..., "2024-03-10 22:00:00"]

after:

["2024-03-10 00:00:00", "2024-03-10 01:00:00", "2024-03-10 03:00:00", ..., "2024-03-10 23:00:00"]

I found this with a property check: every hour bucket ClickHouse can produce for a range must have exactly one label, and no label may exist for an hour that doesn't. I used Tzdata as the oracle for what the local hours of a day really are. Sweeping all 596 zones over every day from 2015 to 2030 gives 3,483,024 (zone, day) pairs; 6,284 of them were wrong before, 0 after. Both directions of the bug show up in every zone that observes DST, and Pacific/Chatham and Antarctica/Casey were also wrong because their transitions aren't on the hour.

time:minute already handles this correctly, and the day/week/month labels are pure Date arithmetic, so this only touches time:hour.

Scope I did not cover: a sub-day window that straddles a backwards transition in a zone whose transitions aren't on the hour can still miss one bucket. In practice that is Pacific/Chatham on its April switch, reachable via "Last 24 Hours" or a custom API range. Fixing it properly needs the bucket-to-range overlap test rather than a naive hour walk, which is a lot more code. Happy to add it if you want it covered.

The three new tests fail on master and pass here. I also ran test/plausible/stats/, main_graph_test.exs and test/plausible_web/controllers/api/external_stats_controller/ before and after: identical results, no new failures.

I used an AI assistant to help write the property-check harness and this patch. I ran everything and checked the results myself.

Tests

• Automated tests have been added

Changelog

• Entry has been added to changelog

Documentation

• This change does not need a documentation update

Dark mode

• This PR does not change the UI

[CLAassistant]

All committers have signed the CLA.

[apata]

Thanks, @jaideeppyne! The buckets we generate app-side (for the response spine) are wrong indeed.

Steps to reproduce

• Visit dashboard of site with time zone that has DST
• Select custom period 1 day that has DST switchover (e.g. Mar 27 2026 Europe/Tallinn)

Expected

• 23 buckets of data
• Time labels: 00:00, 01:00, 02:00, 04:00, 05:00, 06:00, ... 23:00
• All buckets have at least some visitors (for a popular site)

Actual

• 23 buckets of data
• Time labels: 00:00, 01:00, 02:00, 03:00, 04:00, 05:00, ... 22:00
• One bucket value is 0, there's a dip in the chart in the 3rd bucket

Extra info

• Same class of issue for the switch at the other end (e.g. Oct 26 2025 Europe/Tallinn), except the buckets are mislabeled (the hour 3 should appear twice in a row 00:00, 01:00, 02:00, 03:00, 03:00, 04:00, ..., 23:00).
• Same class of issue isn't just with 1-day-long queries, also "Last 7 days" or any other multi-day query that supports time:hour interval. The stored data in continuous in Clickhouse, but it's being mislabeled app side and some of it is dropped.

How does this PR handle queries with an hourly interval that cover days that are 25h long locally?

It seems to me we can't handle them properly without making changes in what we query from Clickhouse (~CH).

If the data from CH is labeled with local timestamps, and local timestamp is the basis for mapping the CH value to the response spine, then for a sparse CH response returning one item local datetime: 2025-10-26 03:00:00, visitors: 10, we don't know which 03:00:00 bucket in the response spine to account those visitors in.

My gut feel is that a full fix would involve

1. fixing the buckets in the response spine, for queries containing 23h long days and 25h long days
2. requesting an extra column from CH that allows us to distinguish what response bucket a value is for, e.g. local datetime: 2025-10-26 03:00:00, datetime: 2025-10-26 00:00:00, visitors: 10, sample below

SELECT toStartOfHour(toTimeZone(timestamp, 'Europe/Tallinn')) h,
       toDateTime(toUnixTimestamp(h)) d,
       uniq(user_id) visitors
FROM events_v2
WHERE site_id = 1
  AND name != 'engagement'
  AND timestamp >= toDateTime('2025-10-25 21:00:00')   -- 2025-10-26 00:00:00 EEST
  AND timestamp <= toDateTime('2025-10-26 21:59:59')   -- 2025-10-26 23:59:59 EET
GROUP BY h
ORDER BY h

[{
  "h": "2025-10-26 03:00:00",
  "d": "2025-10-26 00:00:00",
  "visitors": 219
},
{
  "h": "2025-10-26 03:00:00",
  "d": "2025-10-26 01:00:00",
  "visitors": 196
}]

[jaideeppyne]

Thanks, that reading is right. This PR fixes the app-side hourly spine generation, including not inventing nonexistent spring-forward hours and not stopping at 22:00 on 23-hour days, but it does not fully solve the 25-hour fall-back case if result mapping is keyed only by the local timestamp string.

For a fall-back day with two 03:00:00 local buckets, the response spine can contain two labels, but the CH rows still need an unambiguous bucket identity. If the aggregation result only returns the local hour, a sparse row for 2025-10-26 03:00:00 cannot tell which occurrence it belongs to. So I agree the full fix needs an additional absolute bucket key from CH, for example UTC bucket start / unix timestamp alongside the display-local label, and the response mapper should join on that key rather than only on local datetime text.

I can take either direction from here: broaden this PR to add that CH-side disambiguator and cover both 23h and 25h days end-to-end, or narrow this PR to the spring-forward/dropped-bucket case and leave the fall-back ambiguity for a separate change.

[apata]

@jaideeppyne Thanks for your quick reply!

Of the two, I'd personally prefer the comprehensive solution. I think the ideal solution highlights how important it is to build the response spine correctly app-side if we don't want to use WITH FILL Clickhouse-side. At the moment and even after this PR, it seems to fly under the radar as labels-related, when in reality, as you found, invalid buckets are injected and legitimate buckets returned by CH are dropped in-app.

It's a significant change though and work will likely spill over to comparisons. Comparisons are a bit of a sore spot with difficult decisions. For example, when comparing hourly visitors for the last 7 days with hourly visitors from day -15 to day -8, and one of those periods has more or less hours than the other, how to align the comparison?

TBH I'm actually curious if it'd be possible to get CH to do the spine using WITH FILL. It would eliminate a class of errors completely. If most of the stats requests are not sparse, are increased HTTP payloads between CH / app from use of WITH FILL comparable to requesting an additional disambiguating timestamp column?

[jaideeppyne]

Agreed, I’ll broaden this beyond a label-only fix. I’ll first check whether ClickHouse WITH FILL can generate the hourly spine correctly for the requested timezone without making sparse series/comparison payloads worse. If that does not hold up, I’ll switch the app-side mapping to a stable absolute bucket key and keep the local hour label display-only, with comparison behavior called out explicitly rather than hidden in this PR.