Skip to content

Fix DAG processor treating non-.zip files as DAG bundles (#71125) - #71166

Open
bujjibabukatta wants to merge 1 commit into
apache:mainfrom
bujjibabukatta:fix/#71125
Open

Fix DAG processor treating non-.zip files as DAG bundles (#71125)#71166
bujjibabukatta wants to merge 1 commit into
apache:mainfrom
bujjibabukatta:fix/#71125

Conversation

@bujjibabukatta

Copy link
Copy Markdown
Contributor

DAG processor treats any zip-format file (.jar, .pptx, .docx, .xlsx, etc.) as a potential DAG bundle, not just .zip

Problem

find_dag_file_paths (airflow-core/src/airflow/utils/file.py) decided whether to attempt
DAG discovery on a file using zipfile.is_zipfile() alone — a content sniff for the PK zip
magic bytes, not an extension check. The same unguarded pattern existed in
PythonDagImporter.list_dag_files (airflow-core/src/airflow/dag_processing/importers/python_importer.py),
the newer importer-based discovery path used by DagBag.

Since the zip container format underlies many common file types beyond .zip itself —
.jar, .pptx, .docx, .xlsx, .apk, .epub, .odt, .whl, etc. — any of these
dropped into a DAGs folder (a build artifact, a supporting doc, a packaged dependency) would
pass this check and get opened and scanned via might_contain_dag, purely because it shares
the underlying container format with Airflow's own zipped-DAG-bundle feature. At minimum this
is wasted work on every DAG processor cycle; depending on the archive's contents it can also
produce confusing log noise.

This was reported previously in #45718 with a .pptx file, but that issue was closed as
invalid because the specific symptom reported there turned out to be an unrelated bug. The
underlying zip-detection design issue itself was never fixed.

Fix

Gate both zip-bundle branches on path.suffix == ".zip" in addition to the existing content
sniff, so only files actually named .zip are treated as DAG zip bundles:

  • airflow-core/src/airflow/utils/file.pyfind_dag_file_paths
  • airflow-core/src/airflow/dag_processing/importers/python_importer.pyPythonDagImporter.list_dag_files

I traced every other zipfile.is_zipfile() call site in the DAG processing code
(dag_processing/manager.py's _get_observed_filelocs, python_importer.py's import_file,
and utils/file.py's correct_maybe_zipped/open_maybe_zipped) and confirmed each is only
ever reached after this extension check upstream (or is already gated by a .zip-anchored
regex), so no other locations required changes.

Tests

Added a regression test for each fixed code path, each building a real zip-format file named
.jar (via zipfile.ZipFile, so it genuinely sniffs as a zip) alongside a .py file and a
correctly-named .zip bundle, and asserting the .jar is excluded while the other two are
still discovered:

  • airflow-core/tests/unit/utils/test_file.py::TestListPyFilesPath::test_list_py_file_paths_ignores_non_zip_zip_format_files
  • airflow-core/tests/unit/dag_processing/importers/test_python_importer.py::TestPythonDagImporterListDagFiles::test_list_dag_files_ignores_non_zip_zip_format_files (new file)

Also added airflow-core/newsfragments/71125.bugfix.rst.

closes: #71125


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Claude (Anthropic), used to draft the code fix, the two regression test cases,
this PR description, and the newsfragment, following the guidelines.
All generated code and tests were reviewed and verified by me before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DAG processor treats any zip-format file (.jar, .pptx, .docx, .xlsx, etc.) as a potential DAG bundle, not just .zip

1 participant