Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions docs/adr/0001-interface-contract-closure.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,14 +343,19 @@ not re-validated.

**Context**: Failures were scattered across `CacheErrorHandler`, Bloom and
read-through paths, counted mainly by logs; WARN/ERROR and exception messages
carried raw keys.
carried raw keys. The read-through write-back path may surface a writer
PUT-chain failure, so its metric ownership must remain explicit.

**Decision**: A single internal `CacheFailureReporter` (not public, not in the
allowlist) exposes one metric `resicache.cache.failure` tagged only by finite
enums `operation`, `kind`, `strategy`. `CacheErrorHandler` is the single
count-once exit for all chain failures. WARN/ERROR and typed exception
messages omit the raw key; `cacheName` (config-level, low cardinality) is kept
for correlation. `CacheOperationException` carries no raw-key field/getter.
count-once exit for chain failures, including read-through write-back failures
that go through the writer's PUT chain. `LoaderOrchestrator` does not call the
reporter; it only emits a redacted WARN and returns
`LoadedWithWriteBackFailure`. Exceptions outside that chain are not implicitly
reclassified by this metric. WARN/ERROR and typed exception messages omit the
raw key; `cacheName` (config-level, low cardinality) is kept for correlation.
`CacheOperationException` carries no raw-key field/getter.

Where a diagnostic has no `cacheName` (distributed-lock keys, single-flight
role failures, async early-expiration retries) the raw key is replaced by
Expand All @@ -368,10 +373,13 @@ and the full stack goes to DEBUG, because exception messages can embed the key
stack with the message.

**Consequences**: GET degrade, write fail-fast, REMOVE best-effort and
read-through write-back failures are alertable by bounded tags. The Bloom
filter's own `bloomsift.*` counters and fail-open paths are deliberately
*not* routed here — fail-open is a successful protection behavior, not a
cache-operation failure, so reporting it would corrupt degradation alerts.
writer-PUT-chain-originated read-through write-back failures are alertable by
bounded tags. `LoaderOrchestrator` preserves loaded values and emits only the
redacted tolerated-outcome warning, so the same failure is not counted twice.
The Bloom filter's own `bloomsift.*` counters and fail-open paths are
deliberately *not* routed here — fail-open is a successful protection behavior,
not a cache-operation failure, so reporting it would corrupt degradation
alerts.

**Known limitation**: No per-key alerting; correlation relies on MDC
requestId or the `keyFingerprint` token.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@
* {@code strategy}({@link ErrorStrategy})。禁止 cacheName / key / message tag
* (高基数),WARN/ERROR 日志与异常消息默认不含 raw key(由调用方保证)。
*
* <p>计数去重:同一失败事件只经本类一次上报 — 调用方(链层
* {@code CacheErrorHandler}、read-through write-back)各自在<b>唯一</b>失败出口
* 调用一次,不重复计数。
* <p>计数去重:由 writer PUT chain 产生的写回失败只经本类一次上报 — 链内
* {@code CacheErrorHandler} 是唯一报告出口;{@code LoaderOrchestrator} 只保留脱敏 WARN
* 与 {@code LoadedWithWriteBackFailure},不重复计数。
*
* <p><b>边界(ADR-06)</b>:本指标只统计<b>缓存操作失败</b>(GET degrade / 写 fail-fast /
* REMOVE best-effort / write-back failure)。Bloom 过滤器底层 Redis 位操作的
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,16 +57,19 @@
* </ul>
*
* <p><b>一条协议,两个入口</b>:cache 与 writer 入口都走 {@link #readThrough} —
* 同一套 double-check 语义、同一套写回容错和单点 WARN。两者只在读值表示与 loader
* 异常翻译上不同;sync 路径另外把 cache 协议跑在 {@link SyncSupport} 的分布式锁内。
* 同一套 double-check 语义、同一套写回容错和单点脱敏 WARN。若写回经由 writer 的 PUT
* chain 失败,失败指标已在 {@link CacheErrorHandler} 出口上报一次;本编排器只观察并返回
* tolerated outcome,不再次上报。
* 两者只在读值表示与 loader 异常翻译上不同;sync 路径另外把 cache 协议跑在
* {@link SyncSupport} 的分布式锁内。
*
* <p><b>状态</b>:无可变状态。3 个共享依赖和 3 个 cache-specific callback 由
* {@code RedisProCache} 在构造期一次性绑定(指向 {@code super.createCacheKey} /
* {@code super.get} / {@code this.put});每次调用只需传入 {@code cacheName}、key、
* loader 和 operation。
*
* <p><b>契约保真</b>:异常翻译、键派生({@link CacheKeys})、{@code -1} 永久缓存哨兵、
* null-value 缓存等契约逐字保留;caller-side switch 的 metric 自增保证各路径恰好 1 次 miss 计数。
* caller-side switch 的 miss counter 自增保证各路径恰好 1 次 miss 计数。
*/
@Slf4j
final class LoaderOrchestrator {
Expand Down Expand Up @@ -98,7 +101,8 @@ public record Loaded<T>(@Nullable T value) implements LoadOutcome<T> {
* loader 成功,但缓存写回失败(ADR-02 availability-first)。
*
* <p>{@code value} 仍为 loader 产出的业务值,必须返回给调用方;{@code cause}
* 为写回失败的原始异常,供 caller 记录诊断与失败指标,不覆盖返回值。
* 为写回失败的原始异常,供 caller 记录诊断;若失败来自 writer PUT chain,指标已由
* {@link CacheErrorHandler} 上报一次,本 outcome 不重复计数。
* 锁内 double-check 命中(他线程已加载)不会产生本 outcome —— 该路径无写回。
*/
public record LoadedWithWriteBackFailure<T>(@Nullable T value, Throwable cause)
Expand Down Expand Up @@ -273,8 +277,10 @@ private <T> LoadOutcome<T> executeLoad(
*
* <p>读侧表示({@code R})和业务值表示({@code T})由 caller 显式适配;因此
* {@link Cache.ValueWrapper} 与 writer 的 {@code byte[]} 不需要各自复制协议。
* 写回失败只在此处容忍并发出一次脱敏 WARN;{@link IllegalArgumentException} 保持为
* {@link LoadOutcome.LoadFailed},由 caller 原样抛出。
* 写回若经由 writer 的 PUT chain 失败,链内 {@link CacheErrorHandler} 已完成一次
* failure metric 上报;此处只容忍该失败并发出一次脱敏 WARN,返回
* {@link LoadedWithWriteBackFailure},不再次调用 reporter。{@link IllegalArgumentException}
* 保持为 {@link LoadOutcome.LoadFailed},由 caller 原样抛出。
*
* @param cacheName 缓存名称,仅用于脱敏诊断
* @param read 读原语
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,27 +29,8 @@
@Slf4j
@Configuration(proxyBeanMethods = false)
@Import({
RedisCacheAttributesProjector.class,
SpringCacheableAdapter.class,
CacheHandlerChain.class,
CacheHandlerChainFactory.class,
ChainEngine.class,
ActualCacheHandler.class,
TtlHandler.class,
NullValueEncoder.class,
NullValueHandler.class,
BloomFilterHandler.class,
BloomGate.class,
BloomSupport.class,
SyncLockHandler.class,
SyncLockTimeout.class,
SyncSupport.class,
EarlyExpirationHandler.class,
SecureJacksonSerializerFactory.class,
CacheValueCodec.class,
SerializationPreFlightProbe.class,
SerializerWhitelistStartupGuard.class,
TlsConfigurationValidator.class,
// RedisCacheAutoConfiguration scans the internal runtime package. These two
// configurations are the intentional scan exclusions and remain explicit.
ResolvedMetricsConfiguration.class,
RedisProxyCachingConfiguration.class
})
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import java.time.Duration;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.function.Supplier;
import lombok.extern.slf4j.Slf4j;
import org.slf4j.MDC;
import org.springframework.data.redis.cache.CacheStatistics;
Expand Down Expand Up @@ -140,6 +141,14 @@ public boolean supportsAsyncRetrieve() {
return true;
}

private <T> CompletableFuture<T> submitAsync(Supplier<T> work) {
MethodSnapshot snapshot = operationResolver == null ? null : operationResolver.capture();
Map<String, String> mdcSnapshot = MDC.getCopyOfContextMap();
return CompletableFuture.supplyAsync(
() -> operationResolver == null
? work.get()
: operationResolver.runWithSnapshot(snapshot, mdcSnapshot, work));
}
@Override
@NonNull
public CompletableFuture<byte[]> retrieve(@NonNull String name, @NonNull byte[] key) {
Expand All @@ -150,13 +159,7 @@ public CompletableFuture<byte[]> retrieve(@NonNull String name, @NonNull byte[]
@NonNull
public CompletableFuture<byte[]> retrieve(
@NonNull String name, @NonNull byte[] key, @Nullable Duration ttl) {
MethodSnapshot snapshot = operationResolver == null ? null : operationResolver.capture();
Map<String, String> mdcSnapshot = MDC.getCopyOfContextMap();
return CompletableFuture.supplyAsync(
() -> operationResolver == null
? get(name, key, ttl)
: operationResolver.runWithSnapshot(
snapshot, mdcSnapshot, () -> get(name, key, ttl)));
return submitAsync(() -> get(name, key, ttl));
}

@Override
Expand All @@ -179,17 +182,9 @@ public CompletableFuture<Void> store(
@NonNull byte[] key,
@NonNull byte[] value,
@Nullable Duration ttl) {
MethodSnapshot snapshot = operationResolver == null ? null : operationResolver.capture();
Map<String, String> mdcSnapshot = MDC.getCopyOfContextMap();
return CompletableFuture.runAsync(() -> {
if (operationResolver == null) {
put(name, key, value, ttl);
return;
}
operationResolver.runWithSnapshot(snapshot, mdcSnapshot, () -> {
put(name, key, value, ttl);
return null;
});
return submitAsync(() -> {
put(name, key, value, ttl);
return null;
});
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@



import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
Expand All @@ -27,7 +28,9 @@
* <p>测试配置只替换 RedisConnectionFactory;生产自动配置不依赖
* {@link Primary} 作为用户 Bean 覆盖机制。
*/
@SpringBootTest(classes = {TestApplication.class, RedisDownFaultInjectionIntegrationTest.BrokenRedisConfig.class})
@SpringBootTest(
classes = {TestApplication.class, RedisDownFaultInjectionIntegrationTest.BrokenRedisConfig.class},
properties = "resi-cache.metrics.enabled=true")
@ActiveProfiles({"integration-test", "redis-down-test"})
@Import(TestRedisConfiguration.class)
@DisplayName("Redis 断连故障注入(GET 路径最小切片)")
Expand All @@ -39,6 +42,18 @@ class RedisDownFaultInjectionIntegrationTest extends AbstractRedisIntegrationTes
@Autowired
private org.springframework.cache.CacheManager cacheManager;

@Autowired
private SimpleMeterRegistry meterRegistry;

private double putFailureCount() {
var counter = meterRegistry.find(CacheFailureReporter.METRIC_NAME)
.tag("operation", "PUT")
.tag("kind", "REDIS")
.tag("strategy", "FAIL_FAST")
.counter();
return counter == null ? 0.0 : counter.count();
}

@Test
@DisplayName("RedisDown-6: user-level Cache.get(key, loader) returns loader value on write-back failure")
void redisDown_userLevelGetLoader_loaderValueSurvives() throws Exception {
Expand All @@ -48,11 +63,15 @@ void redisDown_userLevelGetLoader_loaderValueSurvives() throws Exception {
org.springframework.cache.Cache cache = cacheManager.getCache("testCache");
assertThat(cache).isNotNull();

double beforePutFailure = putFailureCount();
String value = cache.get("user-level-loader-key", () -> "business-value");

assertThat(value)
.as("用户级 read-through:loader 成功值必须穿透 Redis-down 写回失败返回")
.isEqualTo("business-value");
assertThat(putFailureCount() - beforePutFailure)
.as("cache read-through 的 PUT chain failure 只能在 CacheErrorHandler 计数一次")
.isEqualTo(1.0);
}

@Test
Expand Down Expand Up @@ -105,6 +124,7 @@ void redisDown_get_degradesGracefully() throws Exception {
void redisDown_writerReadThrough_loaderValueSurvivesWriteBackFailure() {
// writer 级入口(getNativeCache() 可达):缓存读 miss → loader 成功 → 写回失败。
// availability-first:必须返回 loader 值,不得被写回失败覆盖。
double beforePutFailure = putFailureCount();
byte[] result = writer.get(
"testCache",
"fault-injection-loader-key".getBytes(java.nio.charset.StandardCharsets.UTF_8),
Expand All @@ -116,6 +136,9 @@ void redisDown_writerReadThrough_loaderValueSurvivesWriteBackFailure() {
assertThat(new String(result, java.nio.charset.StandardCharsets.UTF_8))
.as("loader 成功值必须穿透写回失败返回")
.isEqualTo("\"loaded-data\"");
assertThat(putFailureCount() - beforePutFailure)
.as("native writer read-through 的 PUT chain failure 只能在 CacheErrorHandler 计数一次")
.isEqualTo(1.0);
}

@Test
Expand Down Expand Up @@ -147,10 +170,17 @@ static class BrokenRedisConfig {
@Bean
@Primary
public RedisConnectionFactory brokenRedisConnectionFactory() {

// 端口 1 — 任何 host 都不会监听(privileged port,典型做法)
LettuceConnectionFactory factory = new LettuceConnectionFactory("127.0.0.1", 1);
factory.setTimeout(2000); // 2s timeout,避免测试 hang
return factory;
}

@Bean
@Primary
public SimpleMeterRegistry meterRegistry() {
return new SimpleMeterRegistry();
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,9 @@ void productionConfiguration_importsInternalConfigurationsExplicitly() {

assertThat(configurationImport).isNotNull();
assertThat(configurationImport.value())
.contains(RedisProxyCachingConfiguration.class, ResolvedMetricsConfiguration.class);
.containsExactlyInAnyOrder(
RedisProxyCachingConfiguration.class,
ResolvedMetricsConfiguration.class);
}

@Test
Expand Down Expand Up @@ -232,6 +234,9 @@ void defaultAssembly_createsProxyAndCacheManager_withoutUserOverrides() throws E
assertThat(context).hasBean("cacheManager");
assertThat(context).hasBean("redisCacheAdvisor");
assertThat(context).hasBean("redisCacheInterceptor");
assertThat(context).hasSingleBean(CacheHandlerChainFactory.class);
assertThat(context).hasSingleBean(ChainEngine.class);
assertThat(context).doesNotHaveBean(CacheHandlerChain.class);
});
}
}
Expand Down
Loading
Loading