Skip to content

feat: distribute bazel JKS truststore via squid-ca-cert (Vault) - #19

Open
ccijunk wants to merge 16 commits into
mainfrom
feat/jks-truststore
Open

feat: distribute bazel JKS truststore via squid-ca-cert (Vault)#19
ccijunk wants to merge 16 commits into
mainfrom
feat/jks-truststore

Conversation

@ccijunk

@ccijunk ccijunk commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Distribution of the Bazel/JVM truststore (squid-bazel-trust.jks) from Vault via the squid-ca-cert SecretDefinition, plus the 0.1.6 cache strategy (per-addressing refresh_pattern rules) and traffic-test tooling fixes.

Changes

  • deploy/chart/templates/secret-definition.yaml: squid-ca-cert keysMap 增加 squid-bazel-trust.jks（Vault property squid_bazel_trust_v3_jks），随 CA 一起分发到 buildkitd / ascend-gha-runners / nv-action / squid
  • deploy/chart/values.yaml: caBundleKey/caPublicKey 对齐线上 v3 名（squid_ca_bundle_v3_pem/squid_ca_v3_pem），新增 caTruststoreKey
  • deploy/chart/templates/configmap.yaml + Chart.yaml: cache strategy 0.1.6（0.1.5 → 0.1.6）
    • immutable 规则只给内容寻址的 A 类制品（.whl/.crate/.deb：10080 100% 525600 ignore-reload override-expire ignore-no-store）
    • 可变 B 类（.pth/.pt/.safetensors、codeload refs/heads、索引页）降级 0 20% 4320（尊重 LM 老化）
    • codeload refs/tags + github releases/download → immutable
    • 移除 squid 7.6 不支持的 ignore-no-cache / override-vary（ERROR/Unknown option）、删除失效 .docker.io 规则、域名转义 \\.pypi\\.org 等、max=525600（消除 cropped 告警）
  • deploy/CACHE-STRATEGY.md（新增）: refresh_pattern 机制、CI 流量 A/B 源站分类（内容寻址 vs 可覆盖）、13 条规则逐条审计、推荐配置形态
  • deploy/tool → traffic-test/tool: 测试目录重命名（typo fix），新增 10-16 用例
  • traffic-test/run-tool-traffic.sh: wait_case 改为等待 Volcano Job phase（vj 分批调度导致窗口污染），修复缺失 -n $NS；新增 wait_pods_done（pods 可能比 vj phase 晚结束）；LOG_DIR 改为可配置环境变量
  • traffic-test/analyze-tool-traffic.py: awk printf %d 将 >2GB 字节计数截断为负值 → %.0f；窗口终点 DONE+180s 捕获分批调度尾部流量；HIT% 与 Prometheus client_out/origin_in 交叉验证
  • traffic-test/tool/01-pip.yaml: set -o pipefail（避免 tail 掩盖失败）、PIP_CERT 去重；jsdelivr CDN 替换为 gh-proxy 获取 requirements.txt
  • traffic-test/tool/03-github.yaml: 移除 github.com 可达性探测（应要求删除，无意义）
  • traffic-test/tool/cachedemo.yaml: hf-mirror.com 加入 NO_PROXY
  • traffic-test/tool/08-bazel.yaml: JKS 经 Vault 同步后是 base64 文本（KV v2 只存字符串），postStart 解码到 /etc/squid-bazel-trust/squid-bazel-trust.jks 再写 .bazelrc
  • traffic-test/run-per-case-wlcb.sh（新增）: wlcb-001 顺序逐 case 监控 wrapper，每 case 独立启动 Prometheus 监控并收集流量 TSV
  • traffic-test/WLCB-001-TRAFFIC-REPORT.md（新增）: wlcb-001 warm cache 16 工具流量模式分析报告，含逐 case 客户端/源站/命中率/模式分类
  • deploy/DEPLOY.md: 更新 §1.2 / §2.3 / §2.5（secret 表、单卷 items 挂载、Vault base64 说明）

关键坑

  1. Vault KV v2 只存字符串，二进制 JKS 写入时自动 base64 → 挂载出来是 base64 文本，需消费端解码一次
  2. K8s secret 卷固定只读（CRI 挂载 ro），解码产物需写到容器层目录或 /tmp
  3. Bazel JVM 忽略 JAVA_TOOL_OPTIONS，trustStore 参数只能走 .bazelrc startup
  4. squid 7.6 移除 ignore-no-cache；override-vary 是未知选项（ERROR）；override-expire/ignore-reload/ignore-no-store 为 legacy WARNING 但有效
  5. ConfigMap 经 subPath 挂载不热更新，配置变更须 rollout restart（kubelet 不会重载 subPath 文件）
  6. Volcano 分批调度（minAvailable=1）：vj 状态 Completed 可能早于最后批次 pods 流量结束（差 1-3 分钟），analyze 窗口需加尾部缓冲，否则 case 窗口内只剩背景流量 → HIT% 假性为 0
  7. 背景 TLS 噪音：未知客户端（源 IP 10.0.0.209/10.0.0.225，不在任何 pod/Service 列表）每 1-5s 对 api.github.com:443 发 CONNECT + TLS 失败（cache.log A000412 = SSL alert bad certificate），每分钟 ~10-20 条 24/7 持续，与 case 流量无关

Test (gy-006, 0.1.6 复测)

case 0.1.4 0.1.6
01 pip — 99.9%（7.5GB out，回源 0.1%）
03 git clone (→gh-proxy) 0% 0.0%（git smart-HTTP pack 为 POST，结构性不可缓存；codeload/releases 规则未被此 case 覆盖，未实测）
05 goproxy 89.7% 100%
08 bazel 94.5% 99.9%（7.4GB HIT）
09 npm 95.4% 99.9%
wget .pth 100% 93.3%（按设计降级，尊重 LM 老化）
  • 08-bazel JKS: DIRECT 26761ms vs SQUID 25546ms，1.0x FASTER（两个变体均 Succeeded）
  • case 01（pip）曾误报 0%：实为 Volcano 分批调度导致 analyze 窗口与真实流量错位（窗口内只有背景 api.github CONNECT），DONE+180s 修复后实测 99.9% HIT
  • case 03 流量经 squid 到 gh-proxy.test.osinfra.cn（03-github.yaml 用 insteadOf 把 github.com 重写到 gh-proxy 镜像），client_out/origin_in 交叉验证全程走代理

wlcb-001 测试 (warm cache, 16 工具)

traffic-test/WLCB-001-TRAFFIC-REPORT.md — 逐 case 流量模式分析。

模式 Cases 特征
平稳流 01-pip, 02-apt, 03-github, 07-cmake, 08-bazel, 09-npm, 11-conda, 12-uv, 15-pnpm, 16-yum 10 个 case，命中率 >91%，源站极低
短时突发 04-curl 命中率 97.5%，源站 1.2 MB/s
低命中直连 06-wget 命中率仅 1.1%，几乎全透传
长尾衰减 05-obsutil 命中率从 96%→2.4%，耗时 760s
下载→编译 10-cargo 650s，前 170s 下载后 480s 编译
快速脉冲 14-git-lfs 35s 完成，命中率 99.7%
跳过 13-huggingface 用户要求跳过

关键发现:

  • 15 个可用 case 中 12 个命中率 >91%，warm cache 效果显著
  • obsutil（OBS 对象存储）和 wget（动态 URL）是最差的缓存场景
  • pip 117s (56 MB/s) vs uv 55s (13.8 MB/s) — uv 虽带宽更低但完成更快

ccijunk added 7 commits August 13, 2026 15:24
@ccijunk
docs(tracfic): record chart 0.1.4 cache-rule optimization + retest re…
ca4a7bb 
…sults

- 3 new refresh_patterns: .crate / .zip / .pth|pt|safetensors
- helm upgrade REVISION 14 (squid-rpardini-0.1.4), ArgoCD ownership + kubelet
  volume-sync gotchas
- retest: cargo 11% -> ~95% hit, wget 94MB .pth TCP_HIT zero-origin

feat(chart): 0.1.4 — force-cache rules for cargo/model-weight traffic

- refresh_pattern for .crate (cargo), .zip, .pth/.pt/.safetensors (model weights)
- measured: cargo hit 11% -> ~95%, wget 94MB .pth TCP_HIT zero-origin (gy-006)
- verified via helm upgrade REVISION 14
@ccijunk
chore: rename branch tracfic-test -> traffic-test (typo fix)
c0d3fa1
@ccijunk
feat(chart): distribute squid_bazel_trust_v3_jks via squid-ca-cert
47e216c 
JKS truststore (Bazel/JDK) fetched from Vault (secrets/data/ascend/ci)
and synced into squid-ca-cert SecretDefinition for CI namespaces
(buildkitd, ascend-gha-runners, nv-action, squid). Align caBundleKey/
caPublicKey to v3 vault properties.
@ccijunk
refactor(chart): hardcode jks secret key name (like squid-ca.pem)
a34ab6d
@ccijunk
refactor(chart): use squid-bazel-trust.jks as jks secret key name
cffb24c
@ccijunk
wip
bdd942f
@ccijunk
wip
c9ae010
@opensourceways-bot

opensourceways-bot commented Aug 13, 2026

Copy link
Copy Markdown

Welcome To opensourceways Community

Hey @ccijunk , thanks for your contribution to the community.

Bot Usage Manual

I'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands.

@opensourceways-bot

opensourceways-bot commented Aug 13, 2026

Copy link
Copy Markdown

CLA Signature Pass

ccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@opensourceways-bot

opensourceways-bot commented Aug 13, 2026

Copy link
Copy Markdown

Linking Issue Notice

@ccijunk , the pull request must be linked to at least one issue.
If an issue has already been linked, but the needs-issue label remains, you can remove the label by commenting /check-issue .

ccijunk added 2 commits August 14, 2026 11:04
@ccijunk
feat(chart): cache strategy 0.1.6 — immutable by addressing, drop inv…
291c9af 
…alid options

- immutable rules only for content-addressed/A-class artifacts (.whl/.crate/.deb)
- .zip/.tar.gz/.pth demoted to LM-aging (0/20%/4320): ref-addressed & overwritable
- codeload refs/tags + github releases/download get immutable; refs/heads short TTL
- remove dead .docker.io rule, unescape domains, max 525960->525600 (no cropped warning)
- add CACHE-STRATEGY.md (full per-rule audit + CI scenario analysis)
@ccijunk
fix(tracfic): wait_case vj-based, analyze awk float, pip pipefail
36d0e0c 
- wait_case: wait on Volcano Job phase (pods scheduled in batches polluted
  subsequent case windows); fix missing -n $NS
- analyze: awk printf %d truncated >2GB byte counters to -2^32 (%.0f)
- 01-pip: add set -o pipefail so pip failures are not masked by tail
@opensourceways-bot

opensourceways-bot commented Aug 14, 2026

Copy link
Copy Markdown

CLA Signature Pass

ccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@ccijunk
fix(tracfic): DONE+180s window tail + document case01 99.9% result
db64909 
- analyze: window ends DONE+180s; Volcano marks Job Completed while
  late-batch pods still stream (case 01 window held only background
  api.github CONNECTs → false 0% HIT)
- run-tool-traffic.sh: wait_pods_done helper (pods may outlive vj phase)
- CACHE-STRATEGY.md §10.1: document Volcano batching, background TLS
  noise (A000412 = bad_certificate alert from unknown 10.0.0.209/225),
  epoch access.log timestamps
- verified case 01 (pip) = 99.9% HIT (7.5GB out, 0.1% origin)
@opensourceways-bot

opensourceways-bot commented Aug 14, 2026

Copy link
Copy Markdown

CLA Signature Pass

ccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@ccijunk
docs(strategy): §3.3 match-order interception — domain rules are meta…
7a9d09c 
…data fallback

Document first-match-wins ordering: extension immutable rules (whl/crate/deb)
intercept artifacts before domain rules, so .pypi.org/.golang.org/.debian.org/
.ubuntu.com actually only fallback to index metadata + unclassified content.
Ordering-sensitive: moving domain rules above extension rules drops .deb/.whl
from immutable to 0 20% 4320.
@opensourceways-bot

opensourceways-bot commented Aug 14, 2026

Copy link
Copy Markdown

CLA Signature Pass

ccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@ccijunk
docs(traffic): cold-cache crash RCA (store_swapout.cc:276 + SFS 92%) …
f356e4b 
…& gitcode 0%

- CACHE-STRATEGY.md §10.2: squid 7.6 stable-build swapout race triggered by
  SWAPFAIL under cold-cache concurrency; SFS shared volume 92% full as amplifier;
  fixes: free registry-cache / collapsed_forwarding / aufs-disks; 7.6 is a stable
  tag (SQUID_7_6), not dev — no upstream fix yet
- TOOL-RESULTS.md: gitcode clone 0% (POST pack, structural); non-git paths
  WAF-blocked (418/403/verification page); single-case cold apt = 65.4% HIT
- run-tool-traffic.sh: wait_pods_done must 'seen' a pod before trusting zero
  running pods (vj may be Completed before any pod is scheduled)
@opensourceways-bot

opensourceways-bot commented Aug 14, 2026

Copy link
Copy Markdown

CLA Signature Pass

ccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@ccijunk
fix(traffic): r6 cold-cache rerun fixes — analyzer clock/IP/log sourcing
85b9682 
- run-tool-traffic.sh wait_pods_done: 'seen' must trigger on ANY pod
  (incl. Succeeded); previous version stuck 30min when pods finished
  before the first poll
- analyze-tool-traffic.py:
  - cluster clock offset (+128s observed) — timeline is local, access.log
    is cluster; without it every window is misaligned
  - drop DONE+180 tail (DONE is now precise; back-to-back cases made
    +180 swallow the next case's traffic) — clamp e to next SUBMIT
  - read access.log from PREVIOUS container stdout (current access.log
    is recreated on restart, ~2min only); pod-IP filtering removed
    (Volcano rebuilds pods, IPs no longer match)
- tool/03-github.yaml: curl raw.githubusercontent.com --max-time 120
  (was hanging forever without timeout)
@opensourceways-bot

opensourceways-bot commented Aug 14, 2026

Copy link
Copy Markdown

CLA Signature Pass

ccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍

@ccijunk
fix(traffic): wlcb-001 warm cache run — gh-proxy, LOG_DIR, rm github …
b76b5f4 
…probe

- 01-pip.yaml: replace jsdelivr CDN with gh-proxy for requirements.txt fetch
- 03-github.yaml: remove github.com reachability probe (meaningless per request)
- cachedemo.yaml: add hf-mirror.com to NO_PROXY
- run-tool-traffic.sh: make LOG_DIR configurable via env var
- run-per-case-wlcb.sh: new wrapper for sequential per-case monitoring on wlcb-001
@opensourceways-bot

opensourceways-bot commented Aug 19, 2026

Copy link
Copy Markdown

CLA Signature Pass

ccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍

ccijunk added 2 commits August 19, 2026 11:13
@ccijunk
docs(traffic): wlcb-001 warm cache traffic pattern analysis report
6e202a3 
16-tool sequential run on wlcb-001 with per-case Prometheus monitoring.
Reports client/origin throughput, hitrate, and pattern classification
(steady, burst, spike, decay) for each tool.
@ccijunk
feat(chart): nodeAffinity/tolerations in chart, wlcb-001/gy-001 value…
cde9564 
…s, huaweicloud mirror

- statefulset.yaml: add nodeAffinity and tolerations support (template)
- values.yaml: add nodeAffinity default (empty), switch alpine mirror to huaweicloud
- values-wlcb-001.yaml: new — wlcb-001 cluster values (amd64, node 192.168.1.49, cache toleration)
- values-gy-001.yaml: new — gy-001 cluster values (amd64, node 192.168.1.191, cache toleration)
@opensourceways-bot

opensourceways-bot commented Aug 19, 2026

Copy link
Copy Markdown

CLA Signature Pass

ccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

needs-issue opensourceways-cla/yes stat/needs-squash

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ccijunk @opensourceways-bot