Consider removing unencrypted rsync server

I have established a new server to replace the final remaining server running on equinix, the 'unencrypted' server. (#3597)

This server is in the OpenJS Azure account where we have a substantial usage credit.

This server is still offering an rsync endpoint, which it rsync's from the www server daily.

The current rsync services have been advertised as "going away" on 2022-01-01, but that clearly hasn't happened yet.

ryan@DESKTOP-P5AM10H:~$ rsync rsync://unencrypted.nodejs.org
nodejs          Node.js releases. This service expires 2022-01-01.
iojs            io.js releases. This service expires 2022-01-01.

Additionally, unencrypted.nodejs.org is listed as a member of two Cloudflare load balancer pools, one of which (nodejs.org) is disabled, and the other "origin.nodejs.org" is still enabled.

I'm ready to switch the DNS for unencrypted so the rsync service continues until we can figure out whether its actually needed anymore and whether or not we have a communication plan to disable that service (if we can).

I'm wondering if the load balancer is even necessary at this point, as I'm pretty sure that's not even being used anymore with the R2/worker based delivery of dist.

Im not going to deploy on a friday before a Holiday weekend, but would like to do this ASAP next week.

The origin.nodejs.org pool is being used by the release worker as a failover, but the unencrypted server being taken out of it during the migration should be fine.

In my opinion, the nodejs.org pool could be removed since it's not being used. If the unencrypted server is to be decommissioned, it won't effect the release worker at all and we could probably remove the origin.nodejs.org pool with it (note this will require a change in the worker to point towards direct.nodejs.org instead of origin.nodejs.org).

This has been deployed to azure, and the load balancers have been changed to use it, as well as the dns for unencrypted.nodejs.org now points at the new server.

That being said, It would be good to remove this service entirely. The rsync endpoint isnt encrypted (thus vulnerable to MITM attacks), and is not well maintained (thus vulnerable to eventual future CVE's).

The services have always been advertised as "going away 01-01-2022". So theres been plenty of warning.

We should flag any previously interested community members about this potential service removal, and schedule some brown outs where we shut it off for a week to see who comes out of the woodwork to complain.

Establishing Thread: #55

This is the issue where support is declared as being dropped Jan 1 2022: #484

In #3960 it was determined that if we made our r2 buckets public, we could recommend mirrors use rclone instead of rsync to maintain mirrors of the dists.

I would like to propose that we do so, opening up the dist-prod/nodejs bucket to allow public read only access.

Im looking for input from the @nodejs/web-infra team as they're much closer to how the Cloudflare release buckets are set up, and I wouldnt want any changes to interfere with anything that exists.

I'm +1 with this. cc @nodejs/web-infra thoughts?

@flakey5

We discussed in Slack, both @flakey5 and I are also +1 here. The only concern is whether regular S3 tooling (e.g., rclone) will work well enough with R2 to allow folks to mirror it.

We're not quite sure what domain to expose for this, though. We could use r2.nodejs.org, but that seems quite provider-specific. dist.nodejs.org would be ideal, but that's already a redirect we probably shouldn't break. I'm currently leaning towards builds.nodejs.org, to align (ish) with unofficial-builds.nodejs.org.

@flakey5

The tooling question turns out to be a sticking point. Because R2 does not have bucket listings, rclone isn't able to retrieve a list of files to mirror. The solution to this is what we already have in place: a worker that provides the listing for us. So I believe that @flakey5 's suggestion in #3960 (comment) is our best bet for a public mirroring situation.

However, A mirror that uses our worker could be resource intensive depending on how its built.

Another option is that we could offer bucket scoped read only API tokens that would allow a mirroring service to use the Cloudflare API to retrieve the listings and mirror them. This strategy has some pros/cons associated with it:

Pros:

rclone would 'just work' with an api token, and wouldn't use any of our worker resources.
We would know who is choosing to mirror the releases, and if they misconfigure (mirror too often, are resource intensive) we'd have somebody we can contact to remediate that situation.

Cons:

mirrors would need permission from the project to get a key-> not as frictionless as an http endpoint, and somebody would have to hand out keys, and keep track of who has been given one, and no guarantees that a key wont be shared externally.
grants read only access to the whole bucket, which might not be what we want (vs a subfolder like prod-dist)

Something to consider with R2 is that it doesn't contain the symlinks, so it cannot be used alone to make a mirror.

I'm of the opinion that the best recommendation we can give going forward is that folks should operate pull-through caches as mirrors, or use the existing access we provide to R2 via the /dist/ worker.

@MattIPv4

In #3960 it was determined that if we made our r2 buckets public, we could recommend mirrors use rclone instead of rsync to maintain mirrors of the dists.

I would like to propose that we do so, opening up the dist-prod/nodejs bucket to allow public read only access.

Im looking for input from the @nodejs/web-infra team as they're much closer to how the Cloudflare release buckets are set up, and I wouldnt want any changes to interfere with anything that exists.

Hi, @MattIPv4. Thanks for coordinating this. I'm curious why the tooling issue could be a blocker for public r2 bucket? From the documentation, I can see that R2 provides an S3-compatible ListObjectsV2 API. It would be enough for downstream mirroring using tools from AWS, i.e. aws s3 sync. I wonder if you may have other concerns about the public r2 bucket.

I'm curious why the tooling issue could be a blocker for public r2 bucket? From the documentation, I can see that R2 provides an S3-compatible ListObjectsV2 API. It would be enough for downstream mirroring using tools from AWS, i.e. aws s3 sync. I wonder if you may have other concerns about the public r2 bucket.

From what I've heard from others that've used R2 buckets, while they are mostly S3-compatible, folks have run into issues when doing things like trying to mirror their full contents, both with compatibility and performance.

As I said before, I think encouraging folks to use pull-through caching is the best path forward, both because of that, also because it ensures we only need to worry about maintaining one surface area for access to the binaries. If we switch implementation again in the future, who knows if it will continue to be S3-compatible, but it is very likely guaranteed that the web interface we expose for downloads would remain the same. It also means we don't need to worry about security or cost concerns with giving folks direct access to the bucket, rather, they're just using the same heavily cached web interface we already have folks downloading via.

The end result for mirrors should be exactly the same, and given we provide index files for all the assets in the bucket, one could even pre-warm a pull-through cache if desired, with the end result being a fully populated mirror just like if you'd mirrored the bucket itself.

I'm curious why the tooling issue could be a blocker for public r2 bucket? From the documentation, I can see that R2 provides an S3-compatible ListObjectsV2 API. It would be enough for downstream mirroring using tools from AWS, i.e. aws s3 sync. I wonder if you may have other concerns about the public r2 bucket.

From what I've heard from others that've used R2 buckets, while they are mostly S3-compatible, folks have run into issues when doing things like trying to mirror their full contents, both with compatibility and performance.

As I said before, I think encouraging folks to use pull-through caching is the best path forward, both because of that, also because it ensures we only need to worry about maintaining one surface area for access to the binaries. If we switch implementation again in the future, who knows if it will continue to be S3-compatible, but it is very likely guaranteed that the web interface we expose for downloads would remain the same. It also means we don't need to worry about security or cost concerns with giving folks direct access to the bucket, rather, they're just using the same heavily cached web interface we already have folks downloading via.

The end result for mirrors should be exactly the same, and given we provide index files for all the assets in the bucket, one could even pre-warm a pull-through cache if desired, with the end result being a fully populated mirror just like if you'd mirrored the bucket itself.

We still choose to make full mirroring. I wonder if you can provide a machine readable file describing the full content and their last modified time so that the sync script will not need crawling the entire repo and checking each file if it is the same as the local version.

Yes, every directory already has a machine-readable file with all the files and their modified timestamps... https://nodejs.org/dist/ just click around from there. There's also https://nodejs.org/dist/index.json and https://nodejs.org/dist/index.tab

The index.html of each directory can be used. A better one can be a recursive index of all the files, so that no crawling into each directory is needed, which can save the requests on both sides. index.json and index.tab only covers all the nodes versions, but not all the files, and thus cannot fit for this need.

@flakey5

I'd defer to others (cc @flakey5) on whether having a single index of every file would be easy to implement/maintain, though it immediately strikes me as something that would be annoying in terms of caching vs. the directory-specific index files where they can be cached much more heavily as most assets in the bucket are immutable.

I'd defer to others (cc @flakey5) on whether having a single index of every file would be easy to implement/maintain, though it immediately strikes me as something that would be annoying in terms of caching vs. the directory-specific index files where they can be cached much more heavily as most assets in the bucket are immutable.

I heard that you have used a Cloud flare worker. I believe that an endpoint an be configured to a worker which then calls ListObjects on the R2 bucket.

That would be very expensive to do on demand, I imagine we'd want an actual file in the R2 bucket that stores that list, so we're not having to repeatedly compute it. But again, I defer to the folks who know more about this, and we can wait for them to chime in.

But, right now, there's nothing stopping folks from operating a pull-through cache, and pre-warming, everything is available via the existing routes we expose for downloads.

@flakey5

I'd defer to others (cc @flakey5) on whether having a single index of every file would be easy to implement/maintain

This is kinda what I needed to do for nodejs/release-cloudflare-worker#754 with the directory cache stuff, though I'm not sure on exposing that publicly since it'd be standardizing the structure of it (fwiw I don't really see it changing much though).

I heard that you have used a Cloud flare worker. I believe that an endpoint an be configured to a worker which then calls ListObjects on the R2 bucket.

This would be horrifically slow due to some limitations on R2's side of things and would definitely need to be cached in a file. A couple months ago I tested this and it took ~10mins to list all of the some 2tb worth of stuff in dist-prod. I was able to get it down to ~1min in a Node env, but those same "optimizations" wouldn't be possible in workers due to the active i/o connection limit.

This would be horrifically slow due to some limitations on R2's side of things and would definitely need to be cached in a file. A couple months ago I tested this and it took ~10mins to list all of the some 2tb worth of stuff in dist-prod. I was able to get it down to ~1min in a Node env, but those same "optimizations" wouldn't be possible in workers due to the active i/o connection limit.

Thanks for looking into this. That's so sad. Then caching the information in an object would be an option.

@shankerwangmiao

@shankerwangmiao The slopacalypse is here and I guided claude into making an rclone script so we can shut down the rsync server. Have a look at #4525 and try out the script to see if it suits your needs.

Navigation Menu

Repository navigation

Consider removing unencrypted rsync server #4085

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Consider removing unencrypted rsync server #4085

Description

Activity

flakey5 commented on May 23, 2025

ryanaslett commented on May 29, 2025

ryanaslett commented on May 29, 2025

ryanaslett commented on May 29, 2025

ryanaslett commented on Nov 21, 2025

ovflowd commented on Nov 21, 2025

MattIPv4 commented on Nov 21, 2025

ryanaslett commented on Nov 24, 2025

targos commented on Nov 25, 2025

MattIPv4 commented on Nov 25, 2025

shankerwangmiao commented on Jan 13, 2026

MattIPv4 commented on Jan 13, 2026

shankerwangmiao commented on Jan 13, 2026

MattIPv4 commented on Jan 13, 2026

shankerwangmiao commented on Jan 13, 2026

MattIPv4 commented on Jan 13, 2026

shankerwangmiao commented on Jan 13, 2026

MattIPv4 commented on Jan 13, 2026

flakey5 commented on Jan 21, 2026

shankerwangmiao commented on Jan 21, 2026

ryanaslett commented on Oct 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions