[reference] Translate search attributes per namespace

[JayChung0258]

What

Search attribute translation now resolves the namespace that owns each attribute and applies that namespace's mapping. Multiple namespaces can use custom search attributes through one proxy.

Why

The config has been keyed by namespaceId since #95, but the runtime ignored the key:

func (s *saTranslator) getNamespaceReqMatcher(namespaceId string) stringMatcher {
	// Placeholder: Just return the first one (only support one namespace mapping)
	for _, matcher := range s.reqMap {
		return matcher

Go randomises map iteration and this runs per request, so two configured namespaces could each be translated with the other's mapping, differently from one request to the next. A panic kept that unreachable, which capped CSA namespaces at one per proxy and forced them to migrate one at a time.

Also silent: two mappings with the same or empty namespaceId overwrote each other, leaving LenNamespaces() == 1, so the panic never fired.

How

• resolveNamespaceID walks up the parent chain to the nearest message that owns a namespace. Upward only, because visit.ValuesUnsafe pops the front of its worklist and swaps in the last element, so traversal order is unspecified.
• Owners are an allowlist of four types, not a NamespaceId name match. StartChildWorkflowExecutionInitiatedEventAttributes holds the child's NamespaceId next to the parent's SearchAttributes, so a name match would apply the wrong mapping. WorkflowExecutionInfo has the same shape via ParentNamespaceId.
• Blob events are a fresh traversal with no view of the enclosing message, so the namespace is resolved at the boundary and carried in. visitDataBlobs takes a blobVisitor closure.
• TranslateResponse(req, resp): GetWorkflowExecutionRawHistoryV2 is in the shipped ACL default and its response carries HistoryBatches but no namespace, so it comes from the paired request. Streams pass nil.
• The translator is stateless (resolvers built once, no receiver assignment), so one instance serves every concurrent stream.

Breaking change: namespaceId is now required

Configs deployed before per-namespace translation omit it, and they work today only because the runtime ignored the key. There is no mapping that means "every namespace", so Validate rejects a blank namespaceId at startup with a message naming the entry.

This affects any deployment, not only the cloud side. searchAttributeTranslation is documented in the chart (charts/s2s-proxy/files/default.yaml:63) as bi-directional, and both the inbound and outbound servers build a translator, so a self-hosted proxy can carry this config too. The chart's example has always included a namespaceId, so configs written against the docs are fine; the exposure is configs written before #95 added the field, or hand-written ones that omitted it.

Before this ships, two things need to happen:

1. The s2c migration tooling emits "namespaceId": "". It has to emit real namespace ids first.
2. Any proxy configured with a blank namespaceId needs its config corrected in the same change as the image.

Both fail loudly at startup rather than silently, which is why this was preferred over grandfathering the empty id.

Other behaviour changes

• Bad config fails at startup naming the entry: missing namespaceId, duplicate namespaceId, duplicate name. Previously one mapping silently overwrote another.
• Add/RemoveSearchAttributesRequest no longer abort the whole message. They also name a field SearchAttributes but hold a map[string]enums.IndexedValueType and a []string, so the default: branch warns, counts, and continues. With no wildcard this branch is unreachable from those two types (neither is enclosed by a namespace owner, so nothing resolves and the field is skipped first), so it is defensive only.
• New counter search_attribute_translation_skipped{reason, message_type}. Not emitted when a namespace resolves but has no mapping, which is normal for any namespace not migrating. No namespace_id label, unbounded cardinality.

Supersedes #96

Same approach to request pairing, but it tracked the last NamespaceId seen while descending. That is order dependent, so it cannot be correct for the multi-namespace case it was written for. It also matched on field name, and looked up real namespace ids against configs whose sole entry is keyed by "", which would have switched translation off silently rather than failing at startup.

Reviewing

git diff main -- ':!*_test.go' is 216 added lines; the rest is tests.

Worth the most attention: are the four owner types complete, and is the skip counter enough to notice if they stop being? They came from enumerating every search attribute and blob site in adminservice, replication, persistence and history.

Known limitation: a cross-namespace child workflow's attributes inside a parent's history blob get the parent's mapping.

Checklist

1. Closes CGSCE-639

2. How was this tested:

Unit tests. The main case puts four replication tasks across three namespaces in one StreamWorkflowReplicationMessagesResponse, two configured with different slots for the same source name and one unconfigured, then asserts each gets its own mapping. Looped 25 times, since one pass can pass by luck. Blobs and IndexedFields maps are built per subtree, because visit.Values skips pointers it has already seen.

Guards cover the child NamespaceId trap, ParentNamespaceId, raw history responses paired and unpaired, an empty-keyed mapping matching nothing, unsupported field types, and a multi-namespace config reaching makeServerOptions. Mutation checked: restoring the first-entry lookup, adding the child event type to the allowlist, or making resolveNamespaceID always return its fallback each turns tests red.

Not done: an end-to-end run with two replicated namespaces.

make generate-test-certs
go test -race -timeout=12m -tags test_dep -count=1 ./...
make lint

3. Any docs updates needed?

Yes, separately. Guidance elsewhere still says CSA namespaces migrate one at a time.

🤖 Generated with Claude Code

[JayChung0258]

Splitting this into a stack of smaller PRs so it is easier to review. Keeping this one open as the place where the whole design is written up in one piece, and closing it when the last stage merges.

Stage 1: #262 (pass a closure to visitDataBlobs)