Skip to content

Add SDC fault confidence levels and reporting - #2

Open
wangxumarshall wants to merge 1 commit into
masterfrom
codex/add-sdc-confidence-enumeration-and-fields
Open

Add SDC fault confidence levels and reporting#2
wangxumarshall wants to merge 1 commit into
masterfrom
codex/add-sdc-confidence-enumeration-and-fields

Conversation

@wangxumarshall

Copy link
Copy Markdown
Owner

Motivation

  • Provide a confidence classification for SDC (Silent Data Corruption) reports so detections can be distinguished by evidence strength and acted on differently.
  • Allow callers to supply a confidence level when logging faults so different detection mechanisms (checksum vs statistical checks) are recorded with appropriate severity.
  • Extend the aggregated report to present per-fault confidence and a confidence-level summary to aid node triage and automated tooling.

Description

  • Add a new enum HPL_T_SDC_CONFIDENCE with HPL_SDC_CONFIRMED, HPL_SDC_SUSPECTED, HPL_SDC_WARNING and a confidence field to HPL_T_SDC_FAULT in hpl/include/hpl_sdc.h.
  • Introduce HPL_sdc_log_fault_ex(...) (accepts confidence) and keep HPL_sdc_log_fault(...) as a wrapper that logs with HPL_SDC_SUSPECTED; update prototypes accordingly.
  • Update hpl/src/sdc/HPL_sdc_report.c to record, gather and print confidence per fault, add HPL_sdc_confidence_str(...), and include a summary counts block for confidence levels in the aggregated report.
  • Add a classifier HPL_sdc_classify_panel_entry(...) in hpl/src/sdc/HPL_sdc_verify.c that maps NaN/Inf to CONFIRMED and finite range overflow to SUSPECTED.
  • Update call sites so detections are logged with an appropriate confidence mapping: broadcast raw checksum mismatch -> CONFIRMED; non-NaN finite/overflow panel entry / floating checksum anomalies -> SUSPECTED (via classifier); panel NaN/Inf -> CONFIRMED (via classifier); panel range overflow -> SUSPECTED; back-solve NaN/Inf -> CONFIRMED; back-solve 6-sigma outliers -> WARNING.

Testing

  • Ran git diff --check to validate whitespace and simple issues and fixed end-of-line normalization; this passed after fixes.
  • Per-file syntax checks were performed with gcc -DHPL_SDC_CHECK -DAdd__ -DF77_INTEGER=int -DStringSunStyle -I/tmp/hpl_fake_mpi -Ihpl/include -Ihpl/include/WSL_SDC_CHECK_ONLY -fsyntax-only <file> for all modified C files, using a minimal fake mpi.h (typedefs for the MPI symbols used); these syntax-only checks succeeded.
  • Attempted a full build with make -C hpl arch=WSL_SDC_CHECK_ONLY, but the build failed in this environment due to platform-specific Make.inc symlink resolution and missing platform Make.inc (environmental issue), so a full compile/link was not completed here.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant