fix(ingestion): report the import environment from the heartbeat asset - #838
Merged
Conversation
Inserting the repository root from automated_ingestion/__init__.py did not fix the ModuleNotFoundError for db in a step process, so the assumption behind that fix was wrong somewhere I cannot see from here. PYTHONPATH=/opt/dagster/app makes the app root importable regardless of how a process was launched, rather than depending on the package having been imported first or on the working directory being on the path. The heartbeat asset now reports cwd, the resolved app root and its contents, whether db and domain are findable, and sys.path. It needs no credentials, so it reports even when everything else fails -- which is what a diagnostic asset is for. If PYTHONPATH does not resolve this, its metadata says why. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 79.27% total — gate is 75%. Coverage for the Python files changed in this PR
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
database_connectivityfails in Dagster+ withModuleNotFoundError: No module named 'db', while the code location loads andingestion_heartbeatruns.Two attempted fixes have not worked — inserting the repository root from
automated_ingestion/__init__.py, and settingPYTHONPATH=/opt/dagster/app.So the assumption behind both is wrong somewhere I cannot see from outside the
container.
What has been ruled out
db/is missing from the image. It is not. I rebuilt the image locallyfrom the same template and context (
context: ./,.dockerignoreexcludesonly
.venv) and/opt/dagster/app/dbis present.find_spec('db')resolves andfrom db.transducer import TransducerObservationsucceeds.db. In the image, an absentdependency names itself —
No module named 'sqlalchemy', not'db'.The remaining explanation is that in the step process,
automated_ingestionresolves from somewhere
dbdoes not. My local check ranpython -c, whichputs the working directory on
sys.path; Dagster runs a console script, whereit does not. That is a real difference and I cannot see which paths the step
process actually has.
What this adds
ingestion_heartbeatnow reportscwd, the resolved app root and itscontents, whether
dbanddomainare findable viafind_spec, and thefull
sys.path. It needs no credentials and no database, so it reports evenwhen everything else fails.
Materializing it answers the question directly instead of another round of
inference.
PYTHONPATHis also added to the setup script'sstoragephase — harmless ifit turns out not to be the cause, and correct regardless.
🤖 Generated with Claude Code