fix: load paths, imports, transpile, and CI permissions by Chessing234 · Pull Request #2124 · MIT-LCP/mimic-code

Chessing234 · 2026-07-23T09:30:53Z

Summary

Quote/escape data dirs and passwords in postgres/sqlite/duckdb loaders
Accept plain .csv (and .csv.gz) in duckdb/sqlite imports; utf-8 transpile writes
sqlfluff annotate no longer fails fork PRs on missing permissions
Concept SQL changes removed — landing in separate PRs (Fix SOFA CV scoring, urine_output_rate diffs, and Elixhauser no_drg #2126 and follow-ups)

Test plan

pytest tests/test_transpile.py
smoke import_duckdb.sh / sqlite import with a spaced datadir

Match kdigo_uo so BigQuery truncation and Postgres fractional diffs no longer diverge on the 6/12h rate windows (MIT-LCP#1549). Co-authored-by: Cursor <cursoragent@cursor.com>

Passwords with quotes and unusual user/db names broke the CREATE USER / DATABASE statements. Co-authored-by: Cursor <cursoragent@cursor.com>

Stop pandas DtypeWarning on chartevents/datetimeevents/ inputevents_cv/noteevents during chunked import (MIT-LCP#1237). Co-authored-by: Cursor <cursoragent@cursor.com>

sqlglot rejects 'D/C''d' as adjacent string literals, breaking BigQuery→postgres/duckdb transpile of this concept. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T09:34:13Z

sqlfluff lint step itself passed; the job fail is the annotate action (permissions on fork PRs).

BigQuery rejects timestamp subtraction with interval compares; use DATETIME_DIFF(..., HOUR) like the rest of the concepts. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T09:47:04Z

also folded in the neuroblock DATETIME_DIFF gap check from #2079.

$$1 expands to the shell PID, so the version arg was never validated. Co-authored-by: Cursor <cursoragent@cursor.com>

Lint already ran; fork tokens cannot create check runs. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T10:16:45Z

CI green on syntax/equivalence/sqlfluff/transpile.

Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T10:37:11Z

also: duckdb .csv load + SOFA rate>0 for CV score 3.

check already handles spaces (MIT-LCP#481/MIT-LCP#2068); load still used unquoted datadir and \cd without quoting, so paths with spaces broke at import. Co-authored-by: Cursor <cursoragent@cursor.com>

C4A (Merkel) sits in the C45–C58 string range; Quan 2005 uses WHO ICD-10 and excludes skin malignancy (MIT-LCP#2017). Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T11:02:17Z

also: charlson C4A (#2017), datadir quotes (#481), utf-8 transpile writes, validate_concepts exact names.

Padded codes break exact and prefix matches the same way as AHRQ (MIT-LCP#1077). Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T12:16:21Z

Also fixed Elixhauser: exclude primary in no_drg, include all diagnoses for DRG version, and RTRIM ICD-9 codes (#1077).

That cutpoint is mmol/L; on MIMIC BUN (mg/dL) it scored normal values as 1 (MIT-LCP#1065). Co-authored-by: Cursor <cursoragent@cursor.com>

Pretty-format column cutting mis-parses table names when rebuilding derived. Co-authored-by: Cursor <cursoragent@cursor.com>

PARSE_DATETIME on a NULL-concatenated string failed rows without a time. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T12:25:23Z

Also fixed LODS BUN 7.50 mg/dL band (#1065), make_concepts drop via bq CSV ls, and null-safe echo charttime parse.

MONTH previously KeyError'd in _SECONDS_PER_UNIT; emit year*12+month. Co-authored-by: Cursor <cursoragent@cursor.com>

Same itemids as IV vitalsign (MIT-LCP#1358): 226329, 227632, 227634. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T12:36:27Z

added: DATETIME_DIFF MONTH transpile, CCO/Arctic Sun temps in III vitals + aline_vitals.

Co-authored-by: Cursor <cursoragent@cursor.com>

SAPS 1984 publishes mmol/L cutpoints; MIMIC labs are mg/dL. Apply PhysioNet Challenge 2012 conversions (glucose x18, BUN x2.8). Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T12:46:54Z

added: echo height PARSE null-guard, aline-aws CCO/Arctic temps, SAPS glucose/BUN mg/dL conversion.

pivoted_oasis reads mimiciii_derived.icustay_hours; make-concepts never created it (or its icustay_times dependency). Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-23T12:56:13Z

also: make-concepts now builds icustay_times/hours (for pivoted_oasis), arterial/central line durations, and neuroblock_dose.

SQL existed but make scripts never ran it (MIT-LCP#830). Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 · 2026-07-24T06:27:30Z

Also wires pivoted_oasis into postgres/duckdb make-concepts (#830).

alistairewj · 2026-07-25T02:46:17Z

there is too much here for one PR - I'm OK with combining together misc doc and syntax fixes into one PR but concept changes are best kept to individual PRs with evidence supporting them

Chessing234 · 2026-07-25T05:49:08Z

Agreed — I’ll split this. Keeping load-path/syntax/transpile fixes here; concept changes (UO rate, vaso pauses, line durations, Elixhauser, vitals, scores, etc.) will go out as separate PRs with the linked issue evidence.

Chessing234 · 2026-07-25T05:54:37Z

Agreed — I’ll split this. Keeping load-path/syntax/transpile fixes here; concept changes (UO rate, vaso pauses, line durations, Elixhauser, vitals, scores, etc.) will go out as separate PRs with the linked issue evidence.

Per maintainer feedback on MIT-LCP#2124, keep load-path / transpile / CI fixes here and move concept logic to individual PRs. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 and others added 4 commits July 23, 2026 12:22

fix: SECOND-based DATETIME_DIFF in urine_output_rate

c06aeec

Match kdigo_uo so BigQuery truncation and Postgres fractional diffs no longer diverge on the 6/12h rate windows (MIT-LCP#1549). Co-authored-by: Cursor <cursoragent@cursor.com>

fix: escape password and quote ids in create_mimic_user

5c49bc4

Passwords with quotes and unusual user/db names broke the CREATE USER / DATABASE statements. Co-authored-by: Cursor <cursoragent@cursor.com>

fix: pin sqlite import dtypes for mixed columns

d4a8305

Stop pandas DtypeWarning on chartevents/datetimeevents/ inputevents_cv/noteevents during chunked import (MIT-LCP#1237). Co-authored-by: Cursor <cursoragent@cursor.com>

fix: avoid escaped quote in neuroblock_dose stopped check

b035858

sqlglot rejects 'D/C''d' as adjacent string literals, breaking BigQuery→postgres/duckdb transpile of this concept. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 mentioned this pull request Jul 23, 2026

fix: emtpy -> empty in mimic-iv postgres load comments #2088

Closed

1 task

fix: use DATETIME_DIFF for neuroblock gap check

24c02b4

BigQuery rejects timestamp subtraction with interval compares; use DATETIME_DIFF(..., HOUR) like the rest of the concepts. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 mentioned this pull request Jul 23, 2026

fix: use DATETIME_DIFF in neuroblock_dose for BigQuery #2079

Closed

1 task

Chessing234 changed the title ~~fix: urine output rate parity, sqlite dtypes, create_user escaping~~ fix: urine output rate parity, sqlite dtypes, create_user, neuroblock Jul 23, 2026

fix: check $1 not PID in copy_concepts script

23f7084

$$1 expands to the shell PID, so the version arg was never validated. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 mentioned this pull request Jul 23, 2026

fix: quote paths in mimic-iii make-concepts.sh #2118

Closed

1 task

ci: dont fail sqlfluff job when annotate lacks fork perms

cb18187

Lint already ran; fork tokens cannot create check runs. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 and others added 2 commits July 23, 2026 13:35

fix: load .csv and .csv.gz in duckdb imports

e560823

Co-authored-by: Cursor <cursoragent@cursor.com>

fix: require epi/norepi rate > 0 for SOFA CV score 3

a2d01c4

Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 changed the title ~~fix: urine output rate parity, sqlite dtypes, create_user, neuroblock~~ fix: UO rate, sqlite dtypes, duckdb csv, SOFA CV, create_user Jul 23, 2026

Chessing234 and others added 4 commits July 23, 2026 13:54

fix: quote datadir for postgres load and wget

1cfb063

check already handles spaces (MIT-LCP#481/MIT-LCP#2068); load still used unquoted datadir and \cd without quoting, so paths with spaces broke at import. Co-authored-by: Cursor <cursoragent@cursor.com>

fix: write transpiled sql as utf-8

81e4155

fix: exact table name match in validate_concepts.sh

b2711aa

fix: exclude ICD-10-CM C4A from Charlson malignancy

8b8833a

C4A (Merkel) sits in the C45–C58 string range; Quan 2005 uses WHO ICD-10 and excludes skin malignancy (MIT-LCP#2017). Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 changed the title ~~fix: UO rate, sqlite dtypes, duckdb csv, SOFA CV, create_user~~ fix: UO rate, charlson C4A, sqlite/duckdb load, SOFA CV Jul 23, 2026

Chessing234 mentioned this pull request Jul 23, 2026

ICD-10-CM codes captured and not captured in charlson.sql #2017

Open

fix: RTRIM ICD-9 codes in Quan Elixhauser

afe7707

Padded codes break exact and prefix matches the same way as AHRQ (MIT-LCP#1077). Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 mentioned this pull request Jul 23, 2026

Spaces in codes in elixhauser_ahrq_v37_no_drg.sql #1077

Open

1 task

Chessing234 and others added 3 commits July 23, 2026 15:24

fix: drop LODS BUN >= 7.50 mg/dL score band

70cfd84

That cutpoint is mmol/L; on MIMIC BUN (mg/dL) it scored normal values as 1 (MIT-LCP#1065). Co-authored-by: Cursor <cursoragent@cursor.com>

fix: drop concept tables via bq ls CSV, not cut -f3

5163678

Pretty-format column cutting mis-parses table names when rebuilding derived. Co-authored-by: Cursor <cursoragent@cursor.com>

fix: skip echo charttime parse when time regex misses

6ce72f9

PARSE_DATETIME on a NULL-concatenated string failed rows without a time. Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 mentioned this pull request Jul 23, 2026

lods.sql BUN value #1065

Open

Chessing234 and others added 3 commits July 23, 2026 15:35

fix: transpile DATETIME_DIFF MONTH for Postgres

d7ec3d5

MONTH previously KeyError'd in _SECONDS_PER_UNIT; emit year*12+month. Co-authored-by: Cursor <cursoragent@cursor.com>

fix: include CCO/Arctic Sun temps in III vitals

1fe4366

Same itemids as IV vitalsign (MIT-LCP#1358): 226329, 227632, 227634. Co-authored-by: Cursor <cursoragent@cursor.com>

fix: include CCO/Arctic Sun temps in aline_vitals

0d80d2b

Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 and others added 3 commits July 23, 2026 15:46

fix: skip echo height PARSE when time regex misses

fc00594

Co-authored-by: Cursor <cursoragent@cursor.com>

fix: include CCO/Arctic Sun temps in aline-aws vitals

b836c5a

Co-authored-by: Cursor <cursoragent@cursor.com>

fix: convert SAPS glucose/BUN thresholds to mg/dL

1cab27c

SAPS 1984 publishes mmol/L cutpoints; MIMIC labs are mg/dL. Apply PhysioNet Challenge 2012 conversions (glucose x18, BUN x2.8). Co-authored-by: Cursor <cursoragent@cursor.com>

fix: build icustay_times/hours before pivoted_oasis

cb5f9ac

pivoted_oasis reads mimiciii_derived.icustay_hours; make-concepts never created it (or its icustay_times dependency). Co-authored-by: Cursor <cursoragent@cursor.com>

Chessing234 and others added 2 commits July 24, 2026 09:25

fix: include pivoted_oasis in postgres/duckdb concept builds

ca766fc

SQL existed but make scripts never ran it (MIT-LCP#830). Co-authored-by: Cursor <cursoragent@cursor.com>

Merge origin/main into fix branch

3a43427

alistairewj closed this Jul 25, 2026

Chessing234 changed the title ~~fix: UO rate, vaso pause gaps, line durations, load paths~~ fix: load paths, imports, transpile, and CI permissions Jul 25, 2026

This was referenced Jul 25, 2026

fix: load paths, imports, transpile, and CI permissions #2127

Closed

fix: quote load paths, imports, and transpile inputs #2128

Open

fix: Charlson age_score half-open decade bands #2129

Merged

Conversation

Chessing234 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 23, 2026

Uh oh!

Chessing234 commented Jul 24, 2026

Uh oh!

alistairewj commented Jul 25, 2026

Uh oh!

Chessing234 commented Jul 25, 2026

Uh oh!

Chessing234 commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Chessing234 commented Jul 23, 2026 •

edited

Loading