fix(frontend): reuse authorization cache without stale grants

docs/design/20261004-authorization-cache-reuse.md

@@ -0,0 +1,93 @@
+# Reuse authorization decisions against current catalog state
+
+Owner: frontend authorization and disttae catalog reads. Follow-up to #29532.
+Initial validation base: `5e965dece9e`; rebased onto `ff53ee2eb0c`.
+
+## Problem and contract
+
+#29532 clears session privileges at every statement to observe remote REVOKE
+and RESTORE. In the TPCC 10/10 comparison of `96196b4a9c` and `ae56c83b62`,
+authorization CPU rises from 23 to 1,025 seconds per five-minute window.
+Repeated catalog SQL adds parsing, planning, compilation and allocation.
+
+Reuse a cached grant only when its catalog dependencies are unchanged at a
+snapshot obtainable by a new authorization transaction. An older data transaction
+snapshot is insufficient. Existing protected-database, view-chain, active-role
+and ownership checks remain the permission decision owners.
+
+## Ownership and validity
+
+Disttae supplies a comparable token for the existing session privilege cache:
+mo_user, mo_role, mo_user_grant, mo_role_grant, mo_role_privs, mo_database and
+mo_tables. The last two cover object ownership and physical catalog table IDs.
+Each entry holds the physical table ID, existing shared-state identity and
+applied logtail watermark. Copies and GC preserve identity; reconstruction does
+not. Tokens retain only tiny identities, never old row/object trees. The applied
+watermark covers row inserts/deletes; the last-flush timestamp does not.
+
+Reject reuse when logtail is not ready. Otherwise acquire a read-only transaction
+through TxnClient.New and close it without workspace, writes or storage commit.
+This reuses existing freshness, admission, cancellation and close ownership;
+it does not claim stronger consistency than current authorization reads.
+Propagate its snapshot lower bound to the session so cache-miss background
+transactions cannot evaluate permissions before the captured token.
+
+Capture only ready subscription states with no pending apply and applied
+watermarks below the exclusive read snapshot. Missing/future catalogs or a
+catalog-generation transition provide no reusable token. Uncertainty clears the
+cache and uses existing authorization SQL; actual snapshot/subscription errors
+propagate. Capture before evaluation: a later commit must invalidate those
+results next time, never stamp them with a newer version.
+
+Role/identity changes, manual clear and cache-mode changes keep their existing
+invalidation. Freshness is checked by the existing privilege and active-role
+consumers, before reading cached decisions. This covers nested SET queries and
+WITH GRANT OPTION without making grant-free controls depend on catalog health.
+Clearing is local; the next authorization revalidates membership in its own RC
+transaction. Ordinary SELECT/UPDATE and bound prepared execution check once;
+transaction controls and Execute shells do not check. Complex multi-object
+operations may require several consuming evaluations.
+
+EXPLAIN authorizes its inner statement. EXPLAIN EXECUTE binding checks the resolved
+AST and plan before releasing the AST. A prepared EXPLAIN EXECUTE rebinds the
+inner mutable handle on execution; its cached query cannot establish that binding.
+Ordinary prepared SELECT/UPDATE retain their plan and compile reuse.
+
+View chains still require metadata checks. Negative lookups allocate nothing;
+positive scopes are capped at 1,024. Table/view scope storage shares one
+implementation; the unused replacement method and unconsumed atomic counters
+are removed.
+
+## Complexity and validation
+
+There is no persistent epoch, wire/config change, broadcaster, worker or second
+permission cache. The hot path exchanges repeated SQL for one read-only snapshot
+and seven catalog lookups. Ordinary data writes preserve the token; unrelated
+metadata changes can conservatively invalidate it. TTL invalidation would permit
+stale grants; an administrator exemption would leave ordinary-user performance
+unfixed and redefine catalog-based authorization.
+
+Main-agent design/self-review: R3 authorization, generation and hot-path changes;
+no subagents. Component tests cover watermark boundaries, pending application,
+reconstruction, failures and scope capacity. Reuse the authenticated two-CN
+fixture for text/SQL-prepared/binary-prepared queries, remote revoke/regrant,
+role switching, old data transactions, DROP/recreate and cancellation. The six
+existing #29399 restore/PITR tests retain their original behavioral oracles.
+
+## Local performance boundary
+
+Same-base one-row workload, three rounds of 1,500 prepared point reads and 300
+BEGIN / SELECT FOR UPDATE / UPDATE / COMMIT transactions. Median microseconds:
+
+| Workload | Per-statement clear | Unsafe reuse control | This change |
+|---|---:|---:|---:|
+| Admin point | 336 | 152 | 149 |
+| Ordinary point | 1,413 | 156 | 173 |
+| Admin transaction | 5,994 | 5,879 | 5,596 |
+| Ordinary transaction | 8,907 | 5,767 | 5,625 |
+
+The unsafe control disables freshness validation only to estimate historical
+cache cost; it is not shipped. Transaction latency is near that control; the
+ordinary point read still costs about 17 microseconds more. These small macOS
+ARM64 measurements, on a machine with other activity, do not prove unchanged
+full TPCC performance. Full TPCC comparison remains required to claim parity.

pkg/frontend/authenticate.go

@@ -74,6 +74,7 @@ import (
 	"github.com/matrixorigin/matrixone/pkg/util/trace"
 	"github.com/matrixorigin/matrixone/pkg/util/trace/impl/motrace"
 	"github.com/matrixorigin/matrixone/pkg/util/trace/impl/motrace/statistic"
+	"github.com/matrixorigin/matrixone/pkg/vm/engine/disttae"
 )
 
 type TenantInfo struct {
@@ -2648,6 +2649,8 @@ const (
 
 // privilegeCache cache privileges on table
 type privilegeCache struct {
+	catalogVersion disttae.PrivilegeCacheVersion
+	cachedScopes   int
 	// For objectType table
 	// For objectType table *, *.*
 	storeForTable [int(privilegeLevelEnd)]btree.Set[PrivilegeType]
@@ -2669,8 +2672,6 @@ type privilegeCache struct {
 	storeForDatabase2 btree.Map[string, *btree.Set[PrivilegeType]]
 	// For objectType account *
 	storeForAccount [int(privilegeLevelEnd)]btree.Set[PrivilegeType]
-	total           atomic.Uint64
-	hit             atomic.Uint64
 
 	// The active primary role is session state, while its grant to the user is
 	// catalog state. Keep the validation in the same cache generation as the
@@ -2738,70 +2739,38 @@ func (pc *privilegeCache) setActiveRoleGrantForGeneration(
 
 // has checks the cache has privilege on a table
 func (pc *privilegeCache) has(objTyp objectType, plt privilegeLevelType, dbName, tableName string, priv PrivilegeType) bool {
-	pc.total.Add(1)
-	privSet := pc.getPrivilegeSet(objTyp, plt, dbName, tableName)
+	privSet := pc.getPrivilegeSet(objTyp, plt, dbName, tableName, false)
 	if privSet != nil {
 		for _, p := range privilegeTypeWithCoveringPrivileges(objTyp, priv) {
 			if privSet.Contains(p) {
-				pc.hit.Add(1)
 				return true
 			}
 		}
 	}
 	return false
 }
 
-func (pc *privilegeCache) getPrivilegeSet(objTyp objectType, plt privilegeLevelType, dbName, tableName string) *btree.Set[PrivilegeType] {
+// Only successful grants allocate a scoped entry. Negative lookups are read-only.
+// Bound the retained scope count independently of catalog/connection lifetime.
+const maxPrivilegeCacheScopes = 1024
+
+func (pc *privilegeCache) getPrivilegeSet(objTyp objectType, plt privilegeLevelType, dbName, tableName string, create bool) *btree.Set[PrivilegeType] {
+	var databases *btree.Map[string, *btree.Set[PrivilegeType]]
+	var relations *btree.Map[string, *btree.Map[string, *btree.Set[PrivilegeType]]]
 	switch objTyp {
-	case objectTypeTable:
-		switch plt {
-		case privilegeLevelStarStar, privilegeLevelStar:
-			return &pc.storeForTable[plt]
-		case privilegeLevelDatabaseStar:
-			dbStore, ok1 := pc.storeForTable2.Get(dbName)
-			if !ok1 {
-				dbStore = &btree.Set[PrivilegeType]{}
-				pc.storeForTable2.Set(dbName, dbStore)
-			}
-			return dbStore
-		case privilegeLevelDatabaseTable, privilegeLevelTable:
-			tableStore, ok1 := pc.storeForTable3.Get(dbName)
-			if !ok1 {
-				tableStore = &btree.Map[string, *btree.Set[PrivilegeType]]{}
-				pc.storeForTable3.Set(dbName, tableStore)
-			}
-			privSet, ok2 := tableStore.Get(tableName)
-			if !ok2 {
-				privSet = &btree.Set[PrivilegeType]{}
-				tableStore.Set(tableName, privSet)
-			}
-			return privSet
-		default:
-			return nil
+	case objectTypeTable, objectTypeView:
+		levels := &pc.storeForTable
+		databases, relations = &pc.storeForTable2, &pc.storeForTable3
+		if objTyp == objectTypeView {
+			levels = &pc.storeForView
+			databases, relations = &pc.storeForView2, &pc.storeForView3
 		}
-	case objectTypeView:
 		switch plt {
 		case privilegeLevelStarStar, privilegeLevelStar:
-			return &pc.storeForView[plt]
+			return &levels[plt]
 		case privilegeLevelDatabaseStar:
-			dbStore, ok1 := pc.storeForView2.Get(dbName)
-			if !ok1 {
-				dbStore = &btree.Set[PrivilegeType]{}
-				pc.storeForView2.Set(dbName, dbStore)
-			}
-			return dbStore
+			relations = nil
 		case privilegeLevelDatabaseTable, privilegeLevelTable:
-			viewStore, ok1 := pc.storeForView3.Get(dbName)
-			if !ok1 {
-				viewStore = &btree.Map[string, *btree.Set[PrivilegeType]]{}
-				pc.storeForView3.Set(dbName, viewStore)
-			}
-			privSet, ok2 := viewStore.Get(tableName)
-			if !ok2 {
-				privSet = &btree.Set[PrivilegeType]{}
-				viewStore.Set(tableName, privSet)
-			}
-			return privSet
 		default:
 			return nil
 		}
@@ -2810,12 +2779,7 @@ func (pc *privilegeCache) getPrivilegeSet(objTyp objectType, plt privilegeLevelT
 		case privilegeLevelStar, privilegeLevelStarStar:
 			return &pc.storeForDatabase[plt]
 		case privilegeLevelDatabase:
-			dbStore, ok1 := pc.storeForDatabase2.Get(dbName)
-			if !ok1 {
-				dbStore = &btree.Set[PrivilegeType]{}
-				pc.storeForDatabase2.Set(dbName, dbStore)
-			}
-			return dbStore
+			databases = &pc.storeForDatabase2
 		default:
 			return nil
 		}
@@ -2824,23 +2788,38 @@ func (pc *privilegeCache) getPrivilegeSet(objTyp objectType, plt privilegeLevelT
 	default:
 		return nil
 	}
-
-}
-
-// set replaces the privileges by new ones
-func (pc *privilegeCache) set(objTyp objectType, plt privilegeLevelType, dbName, tableName string, priv ...PrivilegeType) {
-	privSet := pc.getPrivilegeSet(objTyp, plt, dbName, tableName)
-	if privSet != nil {
-		privSet.Clear()
-		for _, p := range priv {
-			privSet.Insert(p)
+	if create && pc.cachedScopes >= maxPrivilegeCacheScopes {
+		pc.invalidate()
+	}
+	if relations == nil {
+		set, found := databases.Get(dbName)
+		if !found && create {
+			set = &btree.Set[PrivilegeType]{}
+			databases.Set(dbName, set)
+			pc.cachedScopes++
 		}
+		return set
 	}
+	tables, found := relations.Get(dbName)
+	if !found {
+		if !create {
+			return nil
+		}
+		tables = &btree.Map[string, *btree.Set[PrivilegeType]]{}
+		relations.Set(dbName, tables)
+	}
+	set, found := tables.Get(tableName)
+	if !found && create {
+		set = &btree.Set[PrivilegeType]{}
+		tables.Set(tableName, set)
+		pc.cachedScopes++
+	}
+	return set
 }
 
 // add puts the privileges without replacing existed ones
 func (pc *privilegeCache) add(objTyp objectType, plt privilegeLevelType, dbName, tableName string, priv ...PrivilegeType) {
-	privSet := pc.getPrivilegeSet(objTyp, plt, dbName, tableName)
+	privSet := pc.getPrivilegeSet(objTyp, plt, dbName, tableName, true)
 	if privSet != nil {
 		for _, p := range priv {
 			privSet.Insert(p)
@@ -2853,11 +2832,11 @@ func (pc *privilegeCache) invalidate() {
 	if pc == nil {
 		return
 	}
+	pc.catalogVersion = disttae.PrivilegeCacheVersion{}
+	pc.cachedScopes = 0
 	// Advance first so a validation that started in the old generation can
 	// never become visible even if its atomic Store races with the clear below.
 	pc.activeRoleGrantGeneration.Add(1)
-	// total := pc.total.Swap(0)
-	// hit := pc.hit.Swap(0)
 	for i := privilegeLevelStar; i < privilegeLevelEnd; i++ {
 		pc.storeForTable[i].Clear()
 		pc.storeForView[i].Clear()
@@ -2870,13 +2849,6 @@ func (pc *privilegeCache) invalidate() {
 	pc.storeForView3.Clear()
 	pc.storeForDatabase2.Clear()
 	pc.activeRoleGrant.Store(nil)
-	// ratio := float64(0)
-	// if total == 0 {
-	//	ratio = 0
-	// } else {
-	//	ratio = float64(hit) / float64(total)
-	// }
-	// logutil.Debugf("-->hit %d total %d ratio %f", hit, total, ratio)
 }
 
 // verifiedRole holds the role info that has been checked
@@ -8204,6 +8176,11 @@ func determineUserHasPrivilegeSet(ctx context.Context, ses *Session, priv *privi
 		return false, stats, nil
 	}
 
+	// Freshness belongs to authorization consumption, including nested plans.
+	if err = ses.refreshPrivilegeCache(ctx); err != nil {
+		return false, stats, err
+	}
+
 	enableCache, err = privilegeCacheIsEnabled(ctx, ses)
 	if err != nil {
 		return false, stats, err
@@ -8507,6 +8484,11 @@ func validateActiveRoleGrantForAuthorization(
 		return true, stats, nil
 	}
 
+	// Freshness belongs to authorization consumption, including nested plans.
+	if err = ses.refreshPrivilegeCache(ctx); err != nil {
+		return false, stats, err
+	}
+
 	enableCache, err := privilegeCacheIsEnabled(ctx, ses)
 	if err != nil {
 		return false, stats, err