diff --git a/.gitignore b/.gitignore index d083004..eb20d16 100644 --- a/.gitignore +++ b/.gitignore @@ -162,3 +162,11 @@ dist vite.config.js.timestamp-* vite.config.ts.timestamp-* + +.venv/ +.user.yml +.env +profiles.yml +target/ +dbt_packages/ +logs/ diff --git a/AI_ASSIST.md b/AI_ASSIST.md index 76f6a37..7962422 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -4,31 +4,74 @@ Document one place you used an LLM during this assignment. ## The problem - - -TODO +After adding the `dbt_utils.unique_combination_of_columns` test to `fct_daily_borough_stats`, I ran `dbt parse`. The project parsed, but dbt showed a deprecation warning about generic test arguments. I wanted to understand what the warning meant and how to update the YAML syntax correctly. ## The prompt - - -TODO +```text +(.venv) pavel@Pavels-MacBook-Air c55-data-week-10 % ./.venv/bin/dbt parse --profiles-dir . +/Users/pavel/Desktop/data _learn/c55-data-week-10/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 + warnings.warn( +23:12:57 Running with dbt=1.10.22 +23:12:58 Registered adapter: postgres=1.9.1 +23:12:58 Unable to do partial parsing because a project dependency has been added +23:13:00 [WARNING][MissingArgumentsPropertyInGenericTestDeprecation]: Deprecated +functionality +Found top-level arguments to test dbt_utils.unique_combination_of_columns +Arguments to generic tests should be nested under the arguments property. +23:13:00 Performance info: /Users/pavel/Desktop/data _learn/c55-data-week-10/target/perf_info.json +23:13:00 [WARNING][DeprecationsSummary]: Deprecated functionality +Summary of encountered deprecations: +- MissingArgumentsPropertyInGenericTestDeprecation: 1 occurrence +To see all deprecation instances instead of just the first occurrence of each, +run command again with the --show-all-deprecations flag. You may also need to +run with --no-partial-parse` as some deprecations are only encountered during +parsing. +What should I change? +``` ## The response - +The LLM explained that the test logic was correct, but the YAML used an older style for passing arguments to a generic test. It said that in newer dbt style, test parameters should be placed under an explicit `arguments:` key. + +It suggested changing this: + +```yaml +tests: + - dbt_utils.unique_combination_of_columns: + combination_of_columns: + - pickup_borough + - pickup_date +``` + +to this: -TODO +```yaml +tests: + - dbt_utils.unique_combination_of_columns: + arguments: + combination_of_columns: + - pickup_borough + - pickup_date +``` ## Reflection - +I kept the `arguments:` suggestion because it removed the dbt deprecation warning while keeping the same test logic: one row per combination of `pickup_borough` and `pickup_date`. + +I did not change the grain of the mart or the tested columns. I only changed the YAML syntax for the generic test arguments. + +After the change, `dbt parse` ran without the deprecation warning, and the mart tests completed successfully: + +```text +Done. PASS=4 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=4 +``` + +Later, the full project build completed with one expected warning from the singular `avg_tip_pct` test: -TODO +```text +Done. PASS=11 WARN=1 ERROR=0 SKIP=0 NO-OP=0 TOTAL=12 +``` --- diff --git a/docs/lineage.png b/docs/lineage.png new file mode 100644 index 0000000..443b20f Binary files /dev/null and b/docs/lineage.png differ diff --git a/macros/safe_divide.sql b/macros/safe_divide.sql index e30c8fa..4e55732 100644 --- a/macros/safe_divide.sql +++ b/macros/safe_divide.sql @@ -3,8 +3,5 @@ -- Use for tip_pct = tip_amount / fare_amount and similar ratio columns. {% macro safe_divide(numerator, denominator) %} - -- TODO: implement the macro body. - -- Use NULLIF(denominator, 0) to avoid division-by-zero errors. - -- Return only the SQL expression (no SELECT, no semicolon). - NULL + ({{ numerator }} / nullif({{ denominator }}, 0)) {% endmacro %} diff --git a/models/marts/_fct_daily_borough_stats.yml b/models/marts/_fct_daily_borough_stats.yml index 15bba90..9c1efee 100644 --- a/models/marts/_fct_daily_borough_stats.yml +++ b/models/marts/_fct_daily_borough_stats.yml @@ -2,23 +2,27 @@ version: 2 models: - name: fct_daily_borough_stats - description: > - TODO: state the grain (one row per ___), the source lineage - (built from ___ and ___), and at least one known caveat - (e.g. rows dropped in staging, any WARN-severity tests). - # TODO: Task 5 -- add the compound uniqueness test on the mart's primary - # key (pickup_borough, pickup_date). You need the dbt_utils package for - # this: declare it in packages.yml and run `dbt deps` first. + description: "Daily borough-level mart for NYC Green Taxi trips. The grain is one row per pickup borough and pickup date. The model is built from stg_trips joined to stg_zones using pickup_location_id. Trips with null pickup location ids or negative fares are removed in staging, and trips with pickup location ids that do not match a known TLC zone are dropped by the inner join. High average tip percentages are checked by a warning-level singular test because unusual tips can occur in low-volume borough/date combinations." + tests: + - dbt_utils.unique_combination_of_columns: + arguments: + combination_of_columns: + - pickup_borough + - pickup_date columns: - name: pickup_borough - description: "TODO: explain what this column contains and where it comes from" + description: "Borough label of the TLC pickup zone. This comes from stg_zones.borough and can include NYC boroughs as well as labels such as EWR, Unknown, or NaN." + tests: + - not_null - name: pickup_date - description: "TODO: explain units and how it is derived" + description: "Calendar date of the trip pickup, derived from pickup_datetime. Unit: date." + tests: + - not_null - name: trip_count - description: "TODO: explain what is counted (unit: number of trips)" + description: "Number of taxi trips picked up in the borough on the date. Unit: trips." - name: total_fare - description: "TODO: explain units (USD) and what fare_amount represents" + description: "Sum of fare_amount for trips picked up in the borough on the date. Unit: US dollars." - name: avg_tip_pct - description: "TODO: explain the ratio (tip_amount / fare_amount)" + description: "Average of tip_amount divided by fare_amount for trips picked up in the borough on the date. Unit: decimal ratio, where 0.2 means 20%." - name: avg_trip_distance - description: "TODO: explain units (miles, from TLC source data)" + description: "Average trip_distance for trips picked up in the borough on the date. Unit: miles." diff --git a/models/marts/fct_daily_borough_stats.sql b/models/marts/fct_daily_borough_stats.sql index bf47070..21caa49 100644 --- a/models/marts/fct_daily_borough_stats.sql +++ b/models/marts/fct_daily_borough_stats.sql @@ -14,25 +14,15 @@ zones AS ( ) SELECT - -- TODO: join trips to zones on pickup_location_id = location_id. - -- Use an INNER JOIN: a few trips have a pickup_location_id with no matching - -- zone (e.g. 999). INNER JOIN drops those so pickup_borough is never NULL and - -- can serve as part of the mart's primary key (your not_null test needs this). - -- TODO: aggregate to grain (pickup_borough, pickup_date) - -- Required output columns: - -- pickup_borough TEXT - z.borough - -- pickup_date DATE - pickup_datetime::date - -- trip_count BIGINT - count(*) - -- total_fare NUMERIC - sum(fare_amount) - -- avg_tip_pct NUMERIC - avg(tip_pct) - -- avg_trip_distance NUMERIC - avg(trip_distance) - NULL AS pickup_borough, - NULL AS pickup_date, - NULL AS trip_count, - NULL AS total_fare, - NULL AS avg_tip_pct, - NULL AS avg_trip_distance - + z.borough as pickup_borough, + t.pickup_datetime::date as pickup_date, + count(*) as trip_count, + sum(t.fare_amount) as total_fare, + avg(t.tip_pct) as avg_tip_pct, + avg(t.trip_distance) as avg_trip_distance FROM trips t --- TODO: add JOIN to zones here --- TODO: add GROUP BY here +inner join zones z + on t.pickup_location_id = z.location_id +group by + z.borough, + t.pickup_datetime::date diff --git a/models/staging/_sources.yml b/models/staging/_sources.yml index 4bbee9c..17abc34 100644 --- a/models/staging/_sources.yml +++ b/models/staging/_sources.yml @@ -2,9 +2,9 @@ version: 2 sources: - name: nyc_taxi - schema: nyc_taxi # TODO: confirm this matches the schema where raw_trips and raw_zones live + schema: nyc_taxi tables: - name: raw_trips - description: "TODO: one sentence on what this table contains and its grain" + description: "Raw NYC Green Taxi trip records, with one row per taxi trip." - name: raw_zones - description: "TODO: one sentence on what this table contains" + description: "Raw TLC taxi zone lookup table, with one row per location_id." diff --git a/models/staging/_stg_trips.yml b/models/staging/_stg_trips.yml index 14829a8..e97848a 100644 --- a/models/staging/_stg_trips.yml +++ b/models/staging/_stg_trips.yml @@ -2,19 +2,21 @@ version: 2 models: - name: stg_trips - description: "TODO: state the grain (one row per ___) and what source this reads from" + description: "Cleaned trip-level staging model for NYC Green Taxi trips, with one row per trip from raw_trips after removing rows with missing pickup location ids or negative fares." columns: - name: pickup_datetime - description: "TODO" - # TODO: Task 5 -- add not_null tests on every column used as a join or - # group-by key. See the chapter's dbt Tests section for the syntax. + description: "Timestamp when the trip started; used to derive pickup_date in the daily borough mart." + tests: + - not_null - name: pickup_location_id - description: "TODO" + description: "TLC zone identifier where the trip started; used to join trips to stg_zones." + tests: + - not_null - name: fare_amount - description: "TODO" + description: "Fare amount charged for the trip, in US dollars." - name: tip_amount - description: "TODO" + description: "Tip amount paid for the trip, in US dollars." - name: trip_distance - description: "TODO" + description: "Trip distance, in miles." - name: tip_pct - description: "TODO" + description: "Tip amount divided by fare amount; null when fare amount is zero." diff --git a/models/staging/_stg_zones.yml b/models/staging/_stg_zones.yml index c4b785d..d2ca2c8 100644 --- a/models/staging/_stg_zones.yml +++ b/models/staging/_stg_zones.yml @@ -2,11 +2,14 @@ version: 2 models: - name: stg_zones - description: "TODO: state the grain and what source this reads from" + description: "Cleaned TLC taxi zone lookup staging model, with one row per location_id from raw_zones." columns: - name: location_id - description: "TODO" - # TODO: Task 5 -- this column is the join key. Which two generic tests - # guarantee a clean one-to-many join from stg_trips? + description: "Unique TLC zone identifier used to join trips to pickup boroughs." + tests: + - not_null + - unique - name: borough - description: "TODO" + description: "Borough label for the TLC zone, including NYC boroughs plus values such as EWR, Unknown, and NaN." + tests: + - not_null \ No newline at end of file diff --git a/models/staging/stg_trips.sql b/models/staging/stg_trips.sql index b5f7b81..9b84d31 100644 --- a/models/staging/stg_trips.sql +++ b/models/staging/stg_trips.sql @@ -3,11 +3,12 @@ -- Downstream: fct_daily_borough_stats joins this to stg_zones. SELECT - -- TODO: select the columns you need for the mart: - -- pickup_datetime, pickup_location_id, fare_amount, tip_amount, trip_distance - -- - -- TODO: add tip_pct using {{ safe_divide('tip_amount', 'fare_amount') }} - -- - -- TODO: filter out rows where pickup_location_id IS NULL or fare_amount < 0 - + pickup_datetime, + pickup_location_id, + fare_amount, + tip_amount, + trip_distance, + {{ safe_divide('tip_amount', 'fare_amount') }} as tip_pct FROM {{ source('nyc_taxi', 'raw_trips') }} +where pickup_location_id is not null + and fare_amount >= 0 diff --git a/models/staging/stg_zones.sql b/models/staging/stg_zones.sql index cc34d23..b881594 100644 --- a/models/staging/stg_zones.sql +++ b/models/staging/stg_zones.sql @@ -1,7 +1,4 @@ --- Staging model: one row per TLC zone (265 zones). --- Exposes location_id and borough for use as a lookup in the mart. - SELECT - -- TODO: select location_id and borough from {{ source('nyc_taxi', 'raw_zones') }} - + location_id, + borough FROM {{ source('nyc_taxi', 'raw_zones') }} diff --git a/package-lock.yml b/package-lock.yml new file mode 100644 index 0000000..e397fd7 --- /dev/null +++ b/package-lock.yml @@ -0,0 +1,5 @@ +packages: + - name: dbt_utils + package: dbt-labs/dbt_utils + version: 1.3.0 +sha1_hash: 226ae69cdfbc9367e2aa2c472b01f99dbce11de0 diff --git a/packages.yml b/packages.yml index bbb2357..39f82d4 100644 --- a/packages.yml +++ b/packages.yml @@ -1,5 +1,3 @@ -# TODO: Task 5 -- declare the dbt-labs/dbt_utils package here, then run -# `dbt deps` to install it. You need it for the compound uniqueness test -# on the mart. See https://hub.getdbt.com/dbt-labs/dbt_utils/latest/ -# for the package block syntax. -packages: [] +packages: + - package: dbt-labs/dbt_utils + version: 1.3.0 diff --git a/reports/answers.md b/reports/answers.md index 8fb8cf2..3ee78d0 100644 --- a/reports/answers.md +++ b/reports/answers.md @@ -1,18 +1,27 @@ # Business Question Answers -Queries run against `dev_.fct_daily_borough_stats`. +Queries run against `dev_pavel_tisner.fct_daily_borough_stats`. ## Q1: Highest total `total_fare` across the whole loaded dataset **SQL:** ```sql --- TODO: query fct_daily_borough_stats grouped by pickup_borough, sum total_fare, order DESC +select + pickup_borough, + sum(total_fare) as total_fare_across_dataset +from dev_pavel_tisner.fct_daily_borough_stats +group by pickup_borough +order by total_fare_across_dataset desc +limit 1; ``` -**Result:** TODO +**Result:** +| pickup_borough | total_fare_across_dataset | +| --- | ---: | +| Manhattan | 493,955.620... | -**Interpretation:** TODO (one sentence) +**Interpretation:** Manhattan had the highest total fare across the loaded dataset, meaning it generated the most fare revenue among pickup boroughs. --- @@ -21,12 +30,21 @@ Queries run against `dev_.fct_daily_borough_stats`. **SQL:** ```sql --- TODO: query fct_daily_borough_stats grouped by pickup_date, sum trip_count, order DESC LIMIT 1 +select + pickup_date, + sum(trip_count) as overall_trip_count +from dev_pavel_tisner.fct_daily_borough_stats +group by pickup_date +order by overall_trip_count desc +limit 1; ``` -**Result:** TODO +**Result:** +| pickup_date | overall_trip_count | +| --- | ---: | +| 2024-01-17 | 2,221 | -**Interpretation:** TODO (one sentence) +**Interpretation:** January 17, 2024 had the highest overall trip count in the loaded dataset. --- @@ -35,12 +53,25 @@ Queries run against `dev_.fct_daily_borough_stats`. **SQL:** ```sql --- TODO: query fct_daily_borough_stats order by avg_tip_pct DESC LIMIT 5 +select + pickup_borough, + pickup_date, + avg_tip_pct +from dev_pavel_tisner.fct_daily_borough_stats +order by avg_tip_pct desc +limit 5; ``` -**Result:** TODO +**Result:** +| pickup_borough | pickup_date | avg_tip_pct | +| --- | --- | ---: | +| Unknown | 2024-01-30 | 2.500... | +| Unknown | 2024-01-07 | 1.340... | +| Unknown | 2024-01-11 | 1.104... | +| Unknown | 2024-01-16 | 1.000... | +| Unknown | 2024-01-18 | 0.611... | -**Interpretation:** TODO — note whether any avg_tip_pct > 1 rows appear and what causes them +**Interpretation:** The highest average tip percentage was for the `Unknown` borough on 2024-01-30, at about 2.5, meaning tips averaged about 250% of fare amount for that borough/date combination. The `avg_tip_pct > 1` rows are unusual but expected warning-level findings because low-volume or dirty borough buckets can be dominated by a few high-tip trips. --- @@ -49,9 +80,19 @@ Queries run against `dev_.fct_daily_borough_stats`. **SQL:** ```sql --- TODO: use percentile_cont(0.5) WITHIN GROUP (ORDER BY trip_count) filtered by borough +select + pickup_borough, + percentile_cont(0.5) within group (order by trip_count) as median_daily_trip_count +from dev_pavel_tisner.fct_daily_borough_stats +where pickup_borough in ('Manhattan', 'Brooklyn') +group by pickup_borough +order by pickup_borough; ``` -**Result:** TODO +**Result:** +| pickup_borough | median_daily_trip_count | +| --- | ---: | +| Brooklyn | 248.0 | +| Manhattan | 1,169.5 | -**Interpretation:** TODO (one sentence on the ratio) +**Interpretation:** Manhattan’s median daily trip count was much higher than Brooklyn’s, at about 4.7 times Brooklyn’s median daily volume. diff --git a/tests/assert_avg_tip_pct_within_bounds.sql b/tests/assert_avg_tip_pct_within_bounds.sql index b0fab32..901665f 100644 --- a/tests/assert_avg_tip_pct_within_bounds.sql +++ b/tests/assert_avg_tip_pct_within_bounds.sql @@ -2,18 +2,12 @@ -- A tip_pct > 1 means the average tip exceeded the total fare for that cell, -- which is unusual and almost always indicates a small-sample bucket (e.g. the -- Unknown borough) where a few high-tip outliers dominate the average. --- --- Set this test to WARN severity by adding an inline config at the top of this --- file (below these comments): {{ config(severity='warn') }} --- That keeps a few expected Unknown-borough rows from blocking `dbt build`, while --- still surfacing them for your reports/answers.md write-up (the rubric requires --- documenting this finding). Do NOT set a project-level test severity in --- dbt_project.yml: that would also downgrade your not_null and --- unique_combination primary-key tests, which you want to stay at ERROR. --- --- The test passes (no WARN) when zero rows are returned; any returned rows are flagged. --- TODO: write the SELECT here. --- Query {{ ref('fct_daily_borough_stats') }} and return rows where avg_tip_pct > 1. -SELECT NULL AS pickup_borough, NULL AS pickup_date, NULL AS avg_tip_pct -WHERE FALSE -- TODO: replace with the real query +{{ config(severity='warn') }} + +SELECT + pickup_borough, + pickup_date, + avg_tip_pct +from {{ ref('fct_daily_borough_stats') }} +WHERE avg_tip_pct > 1