fix: 新增大 Excel 轻量级解析路径，解决大表格 OOM 关联 issue #53 - #54
Open
ving7176 wants to merge 2 commits into
Open
Conversation
处理超大 Excel（如 33MB XML sheet，74 万单元格）时，openpyxl 全量加载 创建百万级 Cell 对象导致 OOM（12GB+）。 - 新增 _LightweightSheet/_LightCell 等轻量级数据结构，仅存非空单元格 - 新增 SAX 流式解析（iterparse），替代 openpyxl 全量加载 - 连续空行截断：遇到连续 10 行全空即停止解析 - 合并区域按数据边界过滤，丢弃异常合并区域（18.6万 -> 115个） - 多 sheet 独立解析 + 逐 sheet 释放内存 - 从 ZIP 直接提取大 sheet 图片 - gap_tolerance 参数全链路透传（API/CLI/代码） 内存峰值从 12GB+ 降至 2.2GB，大 Excel 从 OOM 崩溃到 16.1s 完成。
- CHANGELOG.md：记录本次修复内容与性能指标 - docs/大Excel_OOM修复案例.md：完整技术案例（根因/方案/验证） - docs/大Excel_OOM修复案例-代码diff.md：改动文件完整 diff
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
描述
处理超大 Excel（如 33MB XML sheet、74 万单元格）时，openpyxl 全量加载创建百万级 Cell 对象，内存占用超过 12GB，进程被 OOM 杀死。本 PR 新增轻量级解析路径（SAX 流式解析），内存峰值从 12GB+ 降至 2.2GB。
Closes #53
根因
load_workbook() 为每个单元格（含空位）创建 Cell 对象 -> 百万级对象 -> 12GB
异常文件含 10 万行+合并区域，拉大洪水填充边界
_find_true_data_bounds 将合并区域纳入边界，max_row=186365，洪水填充遍历 74 万空单元格
解决方案
新增 _LightweightSheet / _LightCell 等轻量级数据结构，仅存非空单元格
新增 _parse_sheet_lightweight：SAX 流式解析（iterparse），替代 openpyxl 全量加载
连续空行截断：遇到连续 10 行全空即停止解析
合并区域按数据边界过滤（18.6 万 -> 115 个）
多 sheet 独立解析 + 逐 sheet 释放内存
从 ZIP 直接提取大 sheet 图片
gap_tolerance 参数全链路透传（API/CLI/代码）