fix(maru_vllm): adapt LMCache kernel integration to relocated KV-type enums

[seohui-XCENA]

Symptom

With current upstream LMCache installed, the first inference request kills the vLLM EngineCore:

File "maru_vllm/connector.py", line 2431, in _packed_load_kernel_ctx
    kv_format = getattr(ops.EngineKVFormat, layout.format_name, None)
AttributeError: 'CudaDeviceOps' object has no attribute 'EngineKVFormat'

The exception escapes save_kv_layer inside the model forward, so the whole engine dies and every request through the frontend returns 500.

Background: why the direct connector touches LMCache at all

MaruKVConnector is a direct vLLM↔maru integration — all cache decisions (chunk keys, placement, metadata, retrieval) are maru's, and no LMCache connector/engine/service ever runs. The only thing borrowed from the lmcache package is its compiled CUDA kernels (multi_layer_kv_transfer / single_layer_kv_transfer), used purely as a library to gather/scatter paged GPU KV blocks to/from contiguous CXL slabs in one kernel launch. This is optional by design: without the kernels the connector uses its per-layer fallback. (Unrelated to maru_lmcache, which is the opposite integration — running the real LMCache connector with maru as its storage backend.)

Root cause

The kernel resolution handled two cases: lmcache not installed (ImportError guard → fallback) and installed-but-missing-a-format (getattr(enum, name, None) → fallback). Upstream LMCache created a third: the KV-type enums (EngineKVFormat, TransferDirection) were relocated out of c_ops into the lmcache.lmcache_native extension (LMCache/LMCache#4453), and the backward-compat forward was removed in #4473 (2026-08-12). lmcache.c_ops now resolves through a device-ops shim that carries the kernels but not the enums, so the outer attribute access raises before the guarded inner lookup runs.

Fix

Two layers, one commit each:

1. Guard the enum lookup itself at both resolution sites (fused single-layer path and the packed load/store kernel ctx), so a c_ops handle without the enum takes the existing per-layer fallback instead of crashing.
2. Resolve the enums from their new home. A _resolve_lmc_ops() helper bundles whichever combination is present — kernels from lmcache.c_ops; enums from c_ops attributes (old builds, including the xcena fork) or from lmcache.lmcache_native (new upstream) — into one namespace with the legacy ops.<name> shape, so all call sites work unchanged against either LMCache generation and the fused path stays active on new upstream.

New config: maru_use_lmcache_kernels

Added to kv_connector_extra_config. Type bool, default true.

Value	Behavior
true (default)	Use LMCache's compiled transfer kernels for the packed GPU↔CXL copies when they can be resolved. If the kernels or enums are unavailable, the connector still degrades to the per-layer paths on its own.
false	Never touch lmcache — force the pure-torch per-layer paths even when lmcache is installed.

Set false to benchmark the fallback paths or to isolate lmcache-related issues. Interaction: the experimental maru_enable_fused_load=true requires the kernels, so with this flag off it is disabled with a warning.

"kv_connector_extra_config": { "maru_use_lmcache_kernels": false }

Validation

• Unit: 9 new tests — shim-crash regressions (fail with the exact production AttributeError pre-fix), resolver across LMCache generations, toggle behavior. Full connector suite passes (172).
• E2E on real hardware (upstream LMCache post-#4473, Dynamo frontend + 2 dynamo.vllm workers, Qwen2.5-32B, CXL /dev/dax pool): previously crashed on the first request. With the fix, cross-instance KV reuse works and the fused kernels engage — the per-load fallback warning is absent, and against the guard-only fallback baseline the medians improved: cold (prefill+store) 1.374s → 1.062s, warm (CXL→GPU load) 0.135s → 0.124s. Fallback-path cross-instance speedup was 10.2× TTFT; the kernel path improves on it mainly on the store side at this KV size (1.1 GB/request).
• Correctness: examples/vllm/p2p_sharing/p2p_example.sh passes on the kernel path (Cache Hit: Yes).

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	6	0	100%
__main__.py	3	3	0%	5, 7–8
allocation_manager.py	129	11	91%	30–31, 44–45, 52, 237–238, 268–271
client.py	187	8	96%	88, 130, 147–148, 311–313, 319
config.py	46	1	98%	72
constants.py	9	0	100%
device_scanner.py	115	5	96%	13–15, 63, 96
handler.py	557	86	85%	153, 168, 179–186, 194–196, 211, 217–219, 227, 238–243, 248, 273, 300–301, 305, 345, 349–350, 355, 377, 384–385, 389–395, 398, 402–404, 420, 422–423, 432, 486, 507, 517–518, 624–626, 633, 755–758, 761, 772–775, 781, 867, 971, 1134–1138, 1144, 1155–1159, 1165, 1244, 1249, 1291
ipc.py	275	2	99%	365, 441
kv_manager.py	108	0	100%
logging_setup.py	19	0	100%
plugin.py	59	3	95%	65–67
protocol.py	233	0	100%
resource_manager_installer.py	103	13	87%	80–86, 167, 169–172, 187
rpc_async_client.py	190	0	100%
rpc_async_server.py	111	0	100%
rpc_client.py	71	0	100%
rpc_client_base.py	106	10	91%	185, 221–222, 233–234, 306–307, 311–312, 375
rpc_handler_mixin.py	106	19	82%	154–156, 159–161, 219–222, 227, 231–235, 256–257, 263
rpc_server.py	64	0	100%
serializer.py	81	0	100%
server.py	170	13	92%	64–65, 73, 168, 172, 243, 247, 265–266, 357–359, 444
stats_manager.py	95	0	100%
types.py	60	1	98%	145
uds_helpers.py	13	0	100%
memory
__init__.py	5	0	100%
allocator.py	55	0	100%
mapper.py	164	17	90%	65–74, 77–80, 293, 342, 376
owned_region_manager.py	101	1	99%	212
types.py	63	0	100%
TOTAL	3325	193	94%

Tests	Skipped	Failures	Errors	Time
724	4 💤	0 ❌	0 🔥	7.939s ⏱️

[kihwan-XCENA]

LGTM!

충돌만 해결하면 될 것 같군요.