Skip to content

Add the ProxyAdminService proto and its generation pipeline - #258

Open
liam-lowe wants to merge 2 commits into
mainfrom
liam-lowe/proxyadmin-service-proto
Open

Add the ProxyAdminService proto and its generation pipeline#258
liam-lowe wants to merge 2 commits into
mainfrom
liam-lowe/proxyadmin-service-proto

Conversation

@liam-lowe

@liam-lowe liam-lowe commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the ProxyAdminService schema, its generated Go stubs, the generation targets, and a CI workflow that keeps them in step. No server or client implementation: this PR is the wire contract only, and nothing calls it yet.

These pieces ship together because the new workflow regenerates api/ and diffs it against the tree. A change to the proto, the buf config, or a make target without the others fails that gate.

The service exposes one RPC, DescribeClusterConnections, which reports a proxy and the cluster connections it has to other proxies.

Routing lives in metadata, not in the request

Requests carry no routing fields. Two pieces of gRPC metadata control how far a call travels. Metadata rather than request fields because an interceptor cannot read a request field without reflection. With fields, every listener's limits would have to be re-checked inside every handler, and any RPC added later would be exposed until someone remembered to add the check.

1. s2s-proxy-scope: how many members of this group answer

flowchart LR
  subgraph M["s2s-proxy-scope: member"]
    direction LR
    O1(["operator"]) -->|call| A1["member-1"]
    A1 -->|"answers for itself"| R1(["one member"])
  end
Loading
flowchart LR
  subgraph G["s2s-proxy-scope: group (the default when absent)"]
    direction LR
    O2(["operator"]) -->|call| B1["member-1"]
    B1 -.->|fan out| B2["member-2"]
    B1 -.->|fan out| B3["member-3"]
    B2 -.->|reply| B1
    B3 -.->|reply| B1
    B1 -->|"one merged answer"| R2(["whole group"])
  end
Loading

Under group, the member that received the call answers for itself and fans out to its siblings, then merges. Each connection's members list then holds one row per member that reported it, and the member that received the call is the row marked self.

2. s2s-proxy-target: hand the question to the counterparty

flowchart LR
  O(["operator"]) -->|"target: prod-migration"| A1["group A<br/>member-1"]
  A1 ==>|"forwarded once,<br/>over that connection's mux"| B1["group B<br/>counterparty"]
  B1 -.->|group scope| B2["member-2"]
  B2 -.-> B1
  B1 ==>|"B's answer"| A1
  A1 --> O
Loading

s2s-proxy-target names a cluster connection. The call is forwarded once to that connection's counterparty, which answers at group scope for its own group. Each listener enforces a ceiling on both headers, so a call travels at most two hops: one forward, then one fan-out. Metadata is not propagated to outgoing calls, so a forwarded or fanned-out request always starts clean.

DescribeClusterConnections

The request is empty. Everything that varies is in metadata.

classDiagram
  class DescribeClusterConnectionsResponse {
    ClusterConnection[] cluster_connections
  }
  class ClusterConnection {
    string name
    ConnectionState state
    int32 mux_sessions_connected
    int32 mux_sessions_total
    int32 mux_sessions_target
    ClusterConnectionMember[] members
  }
  class ClusterConnectionMember {
    Member identity
    ConnectionState state
    int32 mux_sessions_connected
    int32 mux_sessions_total
    int32 mux_sessions_target
  }
  class Member {
    string id
    bool self
    string address
    string version
    Timestamp start_time
  }
  DescribeClusterConnectionsResponse --> ClusterConnection : cluster_connections
  ClusterConnection --> ClusterConnectionMember : members
  ClusterConnectionMember --> Member : identity
Loading

The response is a single list. Each cluster connection carries its members, and each member row nests the member that reported it, so a member's version sits next to its session counts with no join.

A member appears once per connection it reports on, so those per-member facts repeat down the response. There is no top-level description of the member that answered: it is the member marked self.

What the shape gives up

Folding the roster into the connection list removes the only place a member with nothing to report could appear:

  • A member that never answers has no row. It reports no connections, so it appears nowhere. It is not always silent: a discovered member that fails to answer holds a connection that would otherwise read CONNECTED at ERROR. On a connection already reading ERROR it leaves no trace. What is lost is which member, why, and the discovered-against-responding counts. MemberState and the dial error fields are gone with it, having nothing left to describe.
  • There is nowhere to report the discovery provider. It is a fact about the roster rather than about a connection, so MemberDiscovery is gone too.
  • Per-member identity repeats. id, self, address, version and start_time are properties of a member, not of a member-connection pair, and they appear once per connection the member reports on.

In exchange the response has one list instead of two, and no id join.

There is no description of the Temporal cluster a connection fronts. That is ServerDescription under another name, it is not topology, and it would make the proxy originate its own DescribeCluster call to the local frontend. Nothing here populates it. The node that needs the remote cluster's identity already makes that hop and will own the field.

A forwarded response does not mark which of the far side's connections the request arrived over. The two sides name their connections independently, and the chart advises picking a name "not shared by other s2s-proxy deployments", so there is no correspondence to rely on either. A caller making a targeted request is assumed to know what it asked about.

mux_sessions_target sits beside total because they answer different questions. total is what a member holds now. target is what it was configured to hold. A connection holding three of a configured ten reports connected == total, so without target the API cannot express the condition it exists to reveal. The metric cluster_connection_mux_sessions_target already exports that denominator.

ConnectionState is deliberately coarse: UNSPECIFIED, CONNECTED, ERROR. Partial connectivity is ERROR rather than CONNECTED, because three sessions of a configured ten is the condition this endpoint exists to reveal, not a healthy link. UNSPECIFIED means no state was observed, which covers both a field nobody set and a connection this API does not describe, such as one that is not multiplexed. More values will be added; buf.yaml pins breaking: use: FILE, and adding enum values is not a breaking change.

A member that holds zero sessions and a member whose sessions failed both read ERROR. mux_sessions_connected and mux_sessions_total separate them: zero-of-ten is a customer proxy that has not dialled in, three-of-ten is a link that is degraded. Distinguishing "their cluster is down" from "the ACL rejected us" needs a second DescribeCluster hop either way. CONNECTION_STATE_CLOSED can be added later; adding enum values is not a FILE-level break.

No message declares a reserved range. Nothing consumes this wire format yet. ClusterConnection leaves field 5 unused where the cluster description was removed, rather than renumbering every field after it.

buf is vendored

buf is pinned in develop/buf.mod and run through the Go toolchain, the same way golangci-lint and mockgen are pinned in develop/tools.mod. Contributors need no separate install, and CI uses the same pinned version, so CI and a developer's machine cannot disagree about it.

Two things worth knowing:

  • It gets its own module rather than joining tools.mod. buf's dependency graph pulls quic-go and docker, whose versions do not resolve against tools.mod's existing pins. Adding it there produces a module where buf itself fails to compile, and it forces unrelated upgrades on golangci-lint and mockgen. A separate modfile keeps both graphs independent. tools.mod and tools.sum are untouched by this PR.
  • The codegen output does not depend on the buf version. buf.gen.yaml uses remote plugins pinned to protocolbuffers/go:v1.36.11 and grpc/go:v1.5.1, so buf only drives the CLI.

proto-lint and proto-breaking are make targets so the workflow runs what a developer runs.

Testing

  • make generate-proxy-proto reproduces the checked-in api/ exactly, run twice from a cold build cache. That is what the workflow's drift job asserts.
  • make proto-lint passes. The workflow also runs buf breaking against main, skipped on this PR because proto/buf.yaml does not exist on main yet.
  • make test (0 failures), make lint (0 issues) and go build ./... all pass, and golangci-lint still resolves from the unmodified tools.mod.
  • No new tests: the commit adds a schema and generated code, and api/proxyadmin/v1 has no hand-written logic to test.

Worked example

Requested at the default group scope against a three-member group with three cluster connections configured.

Request. Empty. Everything that varies travels in metadata.

grpcurl -plaintext \
  -H 's2s-proxy-scope: group' \
  localhost:6061 \
  temporal.s2sproxy.proxyadmin.v1.ProxyAdminService/DescribeClusterConnections
{}

Response. Every connection in config is listed, whether or not it is up.

{
  "clusterConnections": [
    {
      "name": "prod-migration",
      "state": "CONNECTION_STATE_CONNECTED",
      "muxSessionsConnected": 30,
      "muxSessionsTotal": 30,
      "members": [
        {
          "identity": {
            "id": "s2s-proxy-0",
            "self": true,
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:02Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-1",
            "address": "10.4.1.19:9234",
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:07Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-2",
            "address": "10.4.1.23:9234",
            "version": "1.41.2",
            "startTime": "2026-08-21T18:02:55Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        }
      ]
    },
    {
      "name": "dr-standby",
      "state": "CONNECTION_STATE_ERROR",
      "muxSessionsTotal": 30,
      "members": [
        {
          "identity": {
            "id": "s2s-proxy-0",
            "self": true,
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:02Z"
          },
          "state": "CONNECTION_STATE_ERROR",
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-1",
            "address": "10.4.1.19:9234",
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:07Z"
          },
          "state": "CONNECTION_STATE_ERROR",
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-2",
            "address": "10.4.1.23:9234",
            "version": "1.41.2",
            "startTime": "2026-08-21T18:02:55Z"
          },
          "state": "CONNECTION_STATE_ERROR",
          "muxSessionsTotal": 10
        }
      ]
    },
    {
      "name": "eu-migration",
      "state": "CONNECTION_STATE_ERROR",
      "muxSessionsConnected": 20,
      "muxSessionsTotal": 20,
      "members": [
        {
          "identity": {
            "id": "s2s-proxy-0",
            "self": true,
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:02Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-1",
            "address": "10.4.1.19:9234",
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:07Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-2",
            "address": "10.4.1.23:9234",
            "version": "1.41.2",
            "startTime": "2026-08-21T18:02:55Z"
          },
          "state": "CONNECTION_STATE_ERROR"
        }
      ]
    }
  ]
}

Things to read out of that response:

  • Every configured connection appears, up or not. dr-standby is ERROR with nothing connected, and it is still listed by name with its members. A connection missing from this list is a connection missing from config.
  • eu-migration is ERROR because the members disagree. s2s-proxy-2 has no connection by that name, which is config drift across the group rather than a network problem. The coarse enum cannot yet say which of the two it is.
  • muxSessionsTotal is summed over members that reported the connection. It is 30 for prod-migration across three members, and 20 for eu-migration because only two members have it configured.
  • s2s-proxy-0 is marked self and has no address. It served the call and was never dialled. Every other member was reached over the peer listener at the address shown.
  • s2s-proxy-2 is a version behind. The group is mid-rollout, visible as two distinct versions within one connection's member list.
  • The identity object repeats. Each member's id and version appear once per connection it reports on, three times each here. That is the cost of the response having one list rather than two.
  • Nothing describes the Temporal cluster behind a connection. That belongs to the node that already probes the remote cluster's identity, and no implementation populates it today.
  • Omitted fields are proto3 defaults. muxSessionsConnected is absent rather than 0 on the errored rows.
  • initialFailoverVersion and failoverVersionIncrement are quoted. protojson renders int64 as a string. The int32 counts are plain numbers.

A member that answers nothing appears nowhere in this response. There is no roster to place it in, so a three-member group with one member down is indistinguishable from a two-member group.

At member scope each clusterConnections[].members holds exactly one row, the one marked self.

@liam-lowe
liam-lowe requested a review from a team as a code owner August 20, 2026 20:31
Comment thread .github/workflows/proto.yml Outdated
@liam-lowe liam-lowe changed the title Add the ProxyAdminService proto and its generation pipeline Add the ProxyAdminService proto Aug 20, 2026
@liam-lowe
liam-lowe force-pushed the liam-lowe/proxyadmin-service-proto branch from 279d226 to 3e4f0c0 Compare August 21, 2026 00:55
@liam-lowe liam-lowe changed the title Add the ProxyAdminService proto Add the ProxyAdminService proto and its generation pipeline Aug 21, 2026
@liam-lowe
liam-lowe force-pushed the liam-lowe/proxyadmin-service-proto branch 9 times, most recently from 4a6b862 to b4d1ef4 Compare August 24, 2026 22:10
@liam-lowe liam-lowe changed the title Add the ProxyAdminService proto and its generation pipeline Add the ProxyAdminService proto Aug 24, 2026
@liam-lowe
liam-lowe force-pushed the liam-lowe/proxyadmin-service-proto branch 2 times, most recently from 8ccc7d4 to b6525a5 Compare August 24, 2026 22:49
@liam-lowe liam-lowe changed the title Add the ProxyAdminService proto Add the ProxyAdminService proto and its generation pipeline Aug 24, 2026
@liam-lowe
liam-lowe force-pushed the liam-lowe/proxyadmin-service-proto branch 3 times, most recently from 149245c to 37e2601 Compare August 25, 2026 01:32
The schema, the generated stubs, the make targets and the drift workflow are
one unit: CI regenerates and diffs api/, so a change to any of them without
the others fails the gate.

Requests carry no routing fields. Scope and target travel as gRPC metadata,
because an interceptor cannot read a request field without reflection, so
with fields every listener's limits would have to be re-checked inside every
handler and any RPC added later would be exposed until someone remembered.

The response is one list. Cluster connections carry their members, and each
member row nests the member that reported it, so reading a member's version
next to its session counts needs no join. A member appears once per connection
it reports on and those facts repeat, which is the cost of not having a second
list.

Nothing describes the group as a whole. The responder is the member
marked self.

Session counts carry a target as well as a total. total is what a member holds
now, target is what it was configured to hold. Without target a connection
holding three sessions of a configured ten reports connected == total, which
is the condition this endpoint exists to reveal.

ConnectionState is UNSPECIFIED, CONNECTED and ERROR, and will grow. Partial
connectivity is ERROR rather than CONNECTED: three sessions of a configured
ten is the condition this endpoint exists to reveal, not a healthy link.
UNSPECIFIED means no state was observed, which covers a field nobody set and a
connection this API does not describe, such as one that is not multiplexed.

A member that answered nothing has no row to appear in. It shows up only where
every member that did answer was connected, because a connection that would
otherwise read CONNECTED is held at ERROR instead. On a connection already
reading ERROR it leaves no trace, and a three-member group with one member down
is then indistinguishable from a two-member group.

buf is pinned in develop/buf.mod and run through the Go toolchain rather than
installed separately. The codegen plugins were already remote and version
pinned in buf.gen.yaml, so the buf version only governed the CLI, and CI and a
developer's machine could still disagree about it.

buf gets its own module rather than joining tools.mod. Its dependency graph
pulls quic-go and docker, whose versions do not resolve against tools.mod's
existing pins: adding it there produces a module where buf does not compile.

proto-lint and proto-breaking are make targets so the workflow runs what a
developer runs. The drift step diffs the whole tree rather than api/ alone, so
generation that writes anywhere is caught.

Nothing describes the Temporal cluster a connection fronts. That is
ServerDescription under another name, it is not topology, and it would make the
proxy originate its own DescribeCluster call to the local frontend. Field 5 on
ClusterConnection is left unused rather than renumbered.
@liam-lowe
liam-lowe force-pushed the liam-lowe/proxyadmin-service-proto branch from 37e2601 to 1b9f883 Compare August 25, 2026 01:32
clean: true removes everything under out: before generating, and out: is api/.
The setting is safe today: api/ does not exist on main and is generated in
full. Anything that ever lands there by another route is deleted on the next
generation run without warning.
@liam-lowe
liam-lowe force-pushed the liam-lowe/proxyadmin-service-proto branch from 2b490f6 to 7298853 Compare August 25, 2026 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant