fix(ingestion): install the repository into the Dagster+ image - #840
Merged
Conversation
database_connectivity failed with ModuleNotFoundError: No module named 'db'. The heartbeat diagnostic settled why: inside the running container the working directory is /opt/dagster/app and db/ is present in it, but find_spec reports db and domain as not findable -- that directory is not on sys.path in the process that executes a step. My two previous attempts treated the symptom. Inserting the repository root from automated_ingestion/__init__.py assumed the package's own location could bootstrap the path, and setting PYTHONPATH assumed the variable reached that process. Neither worked, and both left the image depending on path luck. The image now installs the repository via the post-install hook the build template already provides, so db, domain, services, core, and schemas resolve from site-packages regardless of working directory, PYTHONPATH, or which process is importing. --no-deps because the pinned, hashed requirements are already installed and this must not resolve on top of them. Verified in a locally built image: with cwd set to / and no PYTHONPATH, db and domain are findable, db resolves from site-packages, and both import. The sys.path insert is removed rather than left as redundant insurance -- it encoded a theory that turned out to be wrong, and keeping it would suggest the mechanism still matters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 79.26% total — gate is 75%. Coverage for the Python files changed in this PR
|
Contributor
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes
ModuleNotFoundError: No module named 'db'indatabase_connectivity.What the diagnostic showed
Materializing
ingestion_heartbeatinside the container:db/is present in the working directory and still not findable. So/opt/dagster/appis simply not onsys.pathin the process that executes astep — while it is in the process that loads the code location, which is why
the location lists all four assets and
ingestion_heartbeatruns.Why my first two attempts failed
Both treated the symptom, and both were wrong about the mechanism:
automated_ingestion/__init__.py—assumed the package's own location could bootstrap the path.
PYTHONPATH=/opt/dagster/app— assumed the variable reached thatprocess.
Neither worked, and both left the image depending on path luck.
The fix
The build template already runs
dagster_cloud_post_install.shafter therepository is copied and the requirements installed. Using it to
pip install --no-deps .putsdb,domain,services,core, andschemasinsite-packages, where they resolve regardless of working directory,
PYTHONPATH,or which process is importing.
--no-depsbecause the pinned, hashed requirements are already installed andthis must not resolve on top of them. (Appending
.torequirements.txtinstead does not work — pip refuses a directory requirement alongside hashes.)
Verification
Built the image locally and ran under the failing condition — cwd
/, noPYTHONPATH:Also
The
sys.pathinsert is removed rather than kept as redundant insurance. Itencoded a theory that turned out to be wrong, and leaving it in would imply the
mechanism still matters. 42 tests pass; the test that imports
dbfrom anunrelated working directory stays, since it is the behaviour that broke.
🤖 Generated with Claude Code