TiFlash Columnar Hub can segfault in HandleHttpRequestReadyz before TMTContext is initialized

[together-wang]

Bug Report

1. Minimal reproduce step (Required)

Environment:

• TiDBX COLUMNAR topology on aarch64
• Rolling upgrade of compute-tiflash from v26.3.8-nextgen to v26.3.10-nextgen
• Kubernetes readiness probe requests /tiflash/readyz

Steps:

1. Start or roll out a TiFlash Columnar Hub compute node with the target image.
2. Let the status server begin listening while the engine-store server is still initializing.
3. Send GET /tiflash/readyz during this initialization window. The Kubernetes readiness probe can naturally hit this window.
4. Observe that the TiFlash process receives SIGSEGV and the container restarts.

Observed timing from one occurrence:

T+0ms    TiFlash Columnar Hub status server listening
T+26ms   Received signal Segmentation fault(11)
T+120ms  tiflash proxy is initialized
T+189ms  engine-store initialization reaches startProxyService

Relevant crash stack:

Received signal Segmentation fault(11).
Address: 0x2d0
Address not mapped to object.

faultSignalHandler(int, siginfo_t*, void*)
DB::HandleHttpRequestReadyz(EngineStoreServerWrap*, ...)
tiflash_proxy::status_server::handle_request

This is timing-dependent, but it was observed during a real rolling upgrade and caused a new container restart.

2. What did you expect to see? (Required)

Before TMTContext is initialized, /tiflash/readyz should return a non-ready response such as HTTP 500/503. A readiness request must not crash the process.

3. What did you see instead (Required)

The readiness handler dereferenced the engine-store wrapper before tmt was initialized, causing SIGSEGV. Kubernetes restarted the container, and the pod recorded the last termination reason as Error.

The current initialization path appears vulnerable to this ordering:

1. EngineStoreServerWrap::tmt starts as nullptr:
   

   tiflash/dbms/src/Storages/KVStore/FFI/ProxyFFI.h

   Lines 46 to 50 in d89f347

   	struct EngineStoreServerWrap
   	{
   	TMTContext * tmt{nullptr};
   	TiFlashRaftProxyHelper * proxy_helper{nullptr};
   	std::atomic<EngineStoreServerStatus> status{EngineStoreServerStatus::Idle};

2. Columnar Hub starts the status server before waiting for engine-store:
   

   tiflash/contrib/tiflash-columnar-hub/hub-runtime/src/run.rs

   Lines 1489 to 1523 in d89f347

   	let hub_status = Arc::new(AtomicU8::new(RaftProxyStatus::Idle as u8));
   	let hub = ColumnarHub::new(hub_status.clone(), cloud_helper, hub_config_str);
   	let mut ffi_helper = build_hub_ffi_helper(&hub);
   	helper.set_proxy(&mut ffi_helper);
   	let mut status_server = if config.server.status_addr.is_empty() {
   	None
   	} else {
   	let mut server =
   	HubStatusServer::new(security_mgr.clone(), hub_status.clone(), status_config_json)
   	.unwrap_or_else(|err| {
   	panic!(
   	"failed to initialize TiFlash Columnar Hub status server: {}",
   	err
   	)
   	});
   	server
   	.start(config.server.status_addr.clone())
   	.unwrap_or_else(|err| {
   	panic!(
   	"failed to start TiFlash Columnar Hub status server: {}",
   	err
   	)
   	});
   	Some(server)
   	};
   	info!("wait for engine-store server to start");
   	let mut engine_store_status = helper.handle_get_engine_store_server_status();
   	while matches!(engine_store_status, EngineStoreServerStatus::Idle) {
   	thread::sleep(Duration::from_millis(200));
   	engine_store_status = helper.handle_get_engine_store_server_status();
   	}
   	if matches!(engine_store_status, EngineStoreServerStatus::Running) {
   	info!("engine-store server is running");

3. tmt is assigned later in startProxyService:
   

   tiflash/dbms/src/Storages/KVStore/ProxyStateMachine.h

   Lines 346 to 370 in d89f347

   	void startProxyService(TMTContext & tmt_context, const std::optional<raft_serverpb::StoreIdent> & store_ident)
   	{
   	if (!proxy_conf.isProxyRunnable())
   	return;
   	// If a TiFlash starts before any TiKV starts, then the very first Region will be created in TiFlash's proxy and it must be the peer as a leader role.
   	// This conflicts with the assumption that tiflash does not contain any Region leader peer and leads to unexpected errors
   	LOG_INFO(log, "Waiting for TiKV cluster to be bootstrapped");
   	while (!tmt_context.getPDClient()->isClusterBootstrapped())
   	{
   	const int wait_seconds = 3;
   	LOG_ERROR(
   	log,
   	"Waiting for cluster to be bootstrapped, we will sleep for {} seconds and try again.",
   	wait_seconds);
   	::sleep(wait_seconds);
   	}
   	tiflash_instance_wrap.tmt = &tmt_context;
   	LOG_INFO(log, "Let tiflash proxy start all services");
   	// Set tiflash instance status to running, then wait for proxy enter running status
   	// It means that in tiflash-proxy
   	// * rpc server for handling raft command is started
   	// * http status server is started
   	// https://github.com/pingcap/tidb-engine-ext/blob/74d6916e0aee34783cf3835b6fb93d40f32bb889/proxy_components/proxy_server/src/run.rs#L241-L265
   	tiflash_instance_wrap.status = EngineStoreServerStatus::Running;

4. HandleHttpRequestReadyz dereferences server->tmt without a null check:
   

   tiflash/dbms/src/Storages/KVStore/FFI/ProxyFFIStatusService.cpp

   Lines 198 to 245 in d89f347

   	// Return "readiness" status of this tiflash node
   	HttpRequestRes HandleHttpRequestReadyz(
   	EngineStoreServerWrap * server,
   	std::string_view /*path*/,
   	const std::string & api_name,
   	std::string_view query,
   	std::string_view /*body*/)
   	{
   	bool verbose = false;
   	if (query.find("verbose") != std::string_view::npos)
   	verbose = true;
   	bool is_ready = true;
   	std::string response_lines;
   	// Check whether store_status == "running",
   	auto store_status = server->tmt->getStoreStatus(std::memory_order_relaxed);
   	switch (store_status)
   	{
   	case TMTContext::StoreStatus::Running:
   	if (verbose)
   	{
   	response_lines += fmt::format("[+]store_status ok\n");
   	}
   	break;
   	default:
   	if (verbose)
   	{
   	response_lines
   	+= fmt::format("[-]store_status fail: store_status={}\n", magic_enum::enum_name(store_status));
   	}
   	is_ready = false;
   	break;
   	}
   	if (is_ready)
   	{
   	// Return "ok" and 200 status code
   	response_lines += "ok\n";
   	return buildOkResp(api_name, std::move(response_lines));
   	}
   	else
   	{
   	// Not ready for servering requests, return error and 500 status code
   	response_lines += "fail\n";
   	return buildRespWithCode(HttpRequestStatus::InternalError, api_name, std::move(response_lines));
   	}
   	}

Possible fixes include returning not-ready while server == nullptr or server->tmt == nullptr, routing startup probes through the Rust /ready state, or delaying exposure of the FFI readiness endpoint until engine-store initialization completes.

4. What is your TiFlash version? (Required)

Release Version: v26.3.10
Edition: Enterprise
Architecture: aarch64
TiFlash Git Commit: d89f347e5331ad47abe277764f67ff21f6a6976c
Columnar Hub Git Commit: 219b17ddf78dae2b9ae41afeca38dd33f08b9c0c
UTC Build Time: 2026-08-17 08:10:31

[together-wang]

Additional internal references for this occurrence:

• TiKV configuration: tikv_tag=release-nextgen-202603, tikv_version=v26.3.99.
• TiKV was not upgraded in this execution because the effective case argument was upgrade_phase=tiflash; the resolved upgrade step was only tiflash-cn.
• TCMS plan execution (requires internal access): https://tcms.pingcap.net/dashboard/executions/plan/8203806
• Clinic logs and incident window (requires internal access): https://staging-clinic.pingcap.com/portal/#/orgs/10449/clusters/10841429046238564578?from=1787911455&to=1787912598