Skip to content

fix(ingestion): install the repository into the Dagster+ image - #840

Merged
jirhiker merged 1 commit into
stagingfrom
fix/install-repo-in-image
Aug 19, 2026
Merged

fix(ingestion): install the repository into the Dagster+ image#840
jirhiker merged 1 commit into
stagingfrom
fix/install-repo-in-image

Conversation

@jirhiker

Copy link
Copy Markdown
Member

Fixes ModuleNotFoundError: No module named 'db' in database_connectivity.

What the diagnostic showed

Materializing ingestion_heartbeat inside the container:

cwd                /opt/dagster/app
app_root_contents  ... automated_ingestion, db, domain, services ...
db_on_path         false
domain_on_path     false

db/ is present in the working directory and still not findable. So
/opt/dagster/app is simply not on sys.path in the process that executes a
step — while it is in the process that loads the code location, which is why
the location lists all four assets and ingestion_heartbeat runs.

Why my first two attempts failed

Both treated the symptom, and both were wrong about the mechanism:

  1. Inserting the repository root from automated_ingestion/__init__.py
    assumed the package's own location could bootstrap the path.
  2. PYTHONPATH=/opt/dagster/app — assumed the variable reached that
    process.

Neither worked, and both left the image depending on path luck.

The fix

The build template already runs dagster_cloud_post_install.sh after the
repository is copied and the requirements installed. Using it to pip install --no-deps . puts db, domain, services, core, and schemas in
site-packages, where they resolve regardless of working directory, PYTHONPATH,
or which process is importing.

--no-deps because the pinned, hashed requirements are already installed and
this must not resolve on top of them. (Appending . to requirements.txt
instead does not work — pip refuses a directory requirement alongside hashes.)

Verification

Built the image locally and ran under the failing condition — cwd /, no
PYTHONPATH:

db findable: True
domain findable: True
db from: /usr/local/lib/python3.13/site-packages/db/__init__.py
SUCCESS: transducer_observation 15.469751

Also

The sys.path insert is removed rather than kept as redundant insurance. It
encoded a theory that turned out to be wrong, and leaving it in would imply the
mechanism still matters. 42 tests pass; the test that imports db from an
unrelated working directory stays, since it is the behaviour that broke.

🤖 Generated with Claude Code

database_connectivity failed with ModuleNotFoundError: No module named 'db'.
The heartbeat diagnostic settled why: inside the running container the working
directory is /opt/dagster/app and db/ is present in it, but find_spec reports
db and domain as not findable -- that directory is not on sys.path in the
process that executes a step.

My two previous attempts treated the symptom. Inserting the repository root
from automated_ingestion/__init__.py assumed the package's own location could
bootstrap the path, and setting PYTHONPATH assumed the variable reached that
process. Neither worked, and both left the image depending on path luck.

The image now installs the repository via the post-install hook the build
template already provides, so db, domain, services, core, and schemas resolve
from site-packages regardless of working directory, PYTHONPATH, or which
process is importing. --no-deps because the pinned, hashed requirements are
already installed and this must not resolve on top of them.

Verified in a locally built image: with cwd set to / and no PYTHONPATH, db and
domain are findable, db resolves from site-packages, and both import.

The sys.path insert is removed rather than left as redundant insurance -- it
encoded a theory that turned out to be wrong, and keeping it would suggest the
mechanism still matters.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Coverage

79.26% total — gate is 75%.

Coverage for the Python files changed in this PR
Name Stmts Miss Cover Missing
automated_ingestion/__init__.py 0 0 100%
TOTAL 0 0 100%

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Your pull request is automatically being deployed to Dagster Cloud.

Location Status Link Updated
ocotillo-automated-ingestion View in Cloud Aug 19, 2026 at 03:18 AM (UTC)

@jirhiker
jirhiker merged commit 619cdf3 into staging Aug 19, 2026
10 checks passed
@jirhiker
jirhiker deleted the fix/install-repo-in-image branch August 19, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant