diff --git a/.gitignore b/.gitignore index d083004..9c4bee0 100644 --- a/.gitignore +++ b/.gitignore @@ -162,3 +162,11 @@ dist vite.config.js.timestamp-* vite.config.ts.timestamp-* + +# Local secrets and dbt generated files +.env +profiles.yml +dbt_packages/ +target/ +logs/ +grader_output.txt diff --git a/.user.yml b/.user.yml new file mode 100644 index 0000000..f40245d --- /dev/null +++ b/.user.yml @@ -0,0 +1 @@ +id: be40aff5-745e-4ac9-9496-e9d5fb6c1776 diff --git a/AI_ASSIST.md b/AI_ASSIST.md index 76f6a37..e04297d 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -4,33 +4,52 @@ Document one place you used an LLM during this assignment. ## The problem - - -TODO +I used an LLM while debugging my Week 10 dbt assignment after I ran into several errors that blocked the project from running correctly. The main issue was not writing the assignment from scratch, but understanding and fixing setup/runtime problems: VS Code showed false SQL syntax errors for dbt/Jinja syntax, `dbt deps` failed on Windows because `dbt_packages` was locked by another process, YAML parsing failed in `_fct_daily_borough_stats.yml`, and `dbt build` failed because the local `profiles.yml` connection settings were incorrect. ## The prompt - +I asked the LLM several debugging questions while I was trying to get the dbt project to run correctly. The main prompt was: + +```text +I am working on a Week 10 dbt assignment. I already have the dbt models and YAML files, but I am getting several errors while trying to run and validate the project. + +Please help me debug the errors step by step. I want to understand what is wrong, which files need to be fixed, and how to verify that the project works correctly. -TODO +The errors include: +- VS Code shows SQL syntax errors near dbt/Jinja syntax such as {{ ref(...) }}, {{ source(...) }}, and {% macro %}. +- dbt deps fails on Windows with a file-locking error in dbt_packages. +- dbt compile fails with: Env var required but not provided: 'PG_HOST'. +- A YAML file fails with: mapping values are not allowed in this context. +- dbt build fails because of PostgreSQL profile/connection settings. + +Please explain which errors are real dbt problems, which ones are only editor/linter warnings, and give the exact commands I should run to verify the project. +``` ## The response - +The LLM explained that many red VS Code “SQL syntax” errors were false positives because VS Code was parsing dbt/Jinja files as plain PostgreSQL. It advised me to validate the project with dbt compile and dbt build instead of relying only on the Problems panel. + +For the real errors, the LLM identified concrete fixes: -TODO +add dbt_utils correctly in packages.yml; +clean and reinstall dbt_packages after the Windows file-lock error; +fix _fct_daily_borough_stats.yml by using a YAML multiline description with description: |; +use a local profiles.yml only for running dbt, but keep profiles.yml.example for the repo; +make sure the profile points to the correct database, schema, SSL mode, and password environment variable; +run the final validation with: +dbt deps +dbt debug --profiles-dir . +dbt compile --profiles-dir . +dbt build --profiles-dir . --select +fct_daily_borough_stats ## Reflection - +I used the LLM output as debugging guidance, not as a blind code generator. I kept the advice that dbt/Jinja syntax should be checked with dbt compile rather than by the generic SQL linter in VS Code. I also applied the YAML fix using description: |, corrected the package setup, and checked the dbt project with dbt build. -TODO +I did not paste any real password or private connection string into the LLM. I reviewed the suggested changes before applying them and kept the assignment logic focused on the required dbt models, tests, documentation, and business-answer queries. --- > Remember: never paste real connection strings, passwords, or PII into an LLM. > The NYC TLC dataset is public so sample rows are safe here, but practise the habit. +``` diff --git a/docs/lineage.png b/docs/lineage.png new file mode 100644 index 0000000..455c361 Binary files /dev/null and b/docs/lineage.png differ diff --git a/macros/safe_divide.sql b/macros/safe_divide.sql index e30c8fa..2ea1023 100644 --- a/macros/safe_divide.sql +++ b/macros/safe_divide.sql @@ -3,8 +3,5 @@ -- Use for tip_pct = tip_amount / fare_amount and similar ratio columns. {% macro safe_divide(numerator, denominator) %} - -- TODO: implement the macro body. - -- Use NULLIF(denominator, 0) to avoid division-by-zero errors. - -- Return only the SQL expression (no SELECT, no semicolon). - NULL -{% endmacro %} + {{ numerator }} / nullif({{ denominator }}, 0) +{% endmacro %} \ No newline at end of file diff --git a/models/marts/_fct_daily_borough_stats.yml b/models/marts/_fct_daily_borough_stats.yml index 15bba90..e1e0614 100644 --- a/models/marts/_fct_daily_borough_stats.yml +++ b/models/marts/_fct_daily_borough_stats.yml @@ -2,23 +2,55 @@ version: 2 models: - name: fct_daily_borough_stats - description: > - TODO: state the grain (one row per ___), the source lineage - (built from ___ and ___), and at least one known caveat - (e.g. rows dropped in staging, any WARN-severity tests). - # TODO: Task 5 -- add the compound uniqueness test on the mart's primary - # key (pickup_borough, pickup_date). You need the dbt_utils package for - # this: declare it in packages.yml and run `dbt deps` first. + description: | + Daily borough-level mart for NYC green taxi pickup activity. + + Grain: one row per pickup_borough and pickup_date. + + Source lineage: built from stg_trips joined with stg_zones. + stg_trips reads nyc_taxi.raw_trips. + stg_zones reads nyc_taxi.raw_zones. + + Known caveats: stg_trips drops rows with missing pickup_location_id + or negative fare_amount. The mart uses an inner join to drop trips + whose pickup_location_id does not match a known zone. Real TLC labels + such as EWR, Unknown, and dirty NaN borough values remain in the mart. + The avg_tip_pct singular test uses WARN severity because very small + borough/day groups can legitimately exceed 1 due to outliers. + + tests: + - dbt_utils.unique_combination_of_columns: + combination_of_columns: + - pickup_borough + - pickup_date + columns: - name: pickup_borough - description: "TODO: explain what this column contains and where it comes from" + description: Borough label from stg_zones.borough for the trip pickup zone. + tests: + - not_null + - name: pickup_date - description: "TODO: explain units and how it is derived" + description: Calendar date derived from pickup_datetime. Unit is one local pickup day. + tests: + - not_null + - name: trip_count - description: "TODO: explain what is counted (unit: number of trips)" + description: Number of trips picked up in this borough on this date. Unit is trips. + tests: + - not_null + - name: total_fare - description: "TODO: explain units (USD) and what fare_amount represents" + description: Sum of fare_amount for this borough and date. Unit is US dollars. + tests: + - not_null + - name: avg_tip_pct - description: "TODO: explain the ratio (tip_amount / fare_amount)" + description: Average tip percentage for this borough and date. Calculated as tip_amount divided by fare_amount. + tests: + - not_null + - name: avg_trip_distance - description: "TODO: explain units (miles, from TLC source data)" + description: Average trip distance for this borough and date. Unit is miles. + tests: + - not_null \ No newline at end of file diff --git a/models/marts/fct_daily_borough_stats.sql b/models/marts/fct_daily_borough_stats.sql index bf47070..774c650 100644 --- a/models/marts/fct_daily_borough_stats.sql +++ b/models/marts/fct_daily_borough_stats.sql @@ -14,25 +14,15 @@ zones AS ( ) SELECT - -- TODO: join trips to zones on pickup_location_id = location_id. - -- Use an INNER JOIN: a few trips have a pickup_location_id with no matching - -- zone (e.g. 999). INNER JOIN drops those so pickup_borough is never NULL and - -- can serve as part of the mart's primary key (your not_null test needs this). - -- TODO: aggregate to grain (pickup_borough, pickup_date) - -- Required output columns: - -- pickup_borough TEXT - z.borough - -- pickup_date DATE - pickup_datetime::date - -- trip_count BIGINT - count(*) - -- total_fare NUMERIC - sum(fare_amount) - -- avg_tip_pct NUMERIC - avg(tip_pct) - -- avg_trip_distance NUMERIC - avg(trip_distance) - NULL AS pickup_borough, - NULL AS pickup_date, - NULL AS trip_count, - NULL AS total_fare, - NULL AS avg_tip_pct, - NULL AS avg_trip_distance - + z.borough as pickup_borough, + t.pickup_datetime::date as pickup_date, + count(*)::bigint as trip_count, + sum(t.fare_amount)::numeric as total_fare, + avg(t.tip_pct)::numeric as avg_tip_pct, + avg(t.trip_distance)::numeric as avg_trip_distance FROM trips t --- TODO: add JOIN to zones here --- TODO: add GROUP BY here +inner join zones z + on t.pickup_location_id = z.location_id +group by + z.borough, + t.pickup_datetime::date \ No newline at end of file diff --git a/models/staging/_sources.yml b/models/staging/_sources.yml index 4bbee9c..454cb10 100644 --- a/models/staging/_sources.yml +++ b/models/staging/_sources.yml @@ -2,9 +2,9 @@ version: 2 sources: - name: nyc_taxi - schema: nyc_taxi # TODO: confirm this matches the schema where raw_trips and raw_zones live + schema: nyc_taxi tables: - name: raw_trips - description: "TODO: one sentence on what this table contains and its grain" + description: One row per raw NYC green taxi trip record from the loaded January 2024 dataset. - name: raw_zones - description: "TODO: one sentence on what this table contains" + description: One row per TLC taxi zone, mapping location IDs to borough and zone metadata. diff --git a/models/staging/_stg_trips.yml b/models/staging/_stg_trips.yml index 14829a8..fee8189 100644 --- a/models/staging/_stg_trips.yml +++ b/models/staging/_stg_trips.yml @@ -2,19 +2,28 @@ version: 2 models: - name: stg_trips - description: "TODO: state the grain (one row per ___) and what source this reads from" + description: One row per cleaned NYC green taxi trip, read from nyc_taxi.raw_trips. Rows with missing pickup_location_id or negative fare_amount are removed before aggregation. columns: - name: pickup_datetime - description: "TODO" - # TODO: Task 5 -- add not_null tests on every column used as a join or - # group-by key. See the chapter's dbt Tests section for the syntax. + description: Wall-clock timestamp when the trip started; used to derive pickup_date in the mart. + tests: + - not_null - name: pickup_location_id - description: "TODO" + description: TLC pickup zone identifier; join key to stg_zones.location_id. + tests: + - not_null + - relationships: + to: ref('stg_zones') + field: location_id + severity: warn - name: fare_amount - description: "TODO" + description: Metered trip fare in USD before tip; negative fares are filtered out in this staging model. + tests: + - not_null - name: tip_amount - description: "TODO" + description: Tip amount in USD recorded for the trip. - name: trip_distance - description: "TODO" + description: Trip distance in miles recorded for the trip. - name: tip_pct - description: "TODO" + description: Ratio tip_amount / fare_amount; NULL when fare_amount is zero. + diff --git a/models/staging/_stg_zones.yml b/models/staging/_stg_zones.yml index c4b785d..be92701 100644 --- a/models/staging/_stg_zones.yml +++ b/models/staging/_stg_zones.yml @@ -2,11 +2,15 @@ version: 2 models: - name: stg_zones - description: "TODO: state the grain and what source this reads from" + description: One row per TLC taxi zone from nyc_taxi.raw_zones, used as the borough lookup for pickup_location_id. columns: - name: location_id - description: "TODO" - # TODO: Task 5 -- this column is the join key. Which two generic tests - # guarantee a clean one-to-many join from stg_trips? + description: TLC taxi zone identifier; primary key of the zone lookup and join target for stg_trips.pickup_location_id. + tests: + - unique + - not_null - name: borough - description: "TODO" + description: Borough or special TLC borough label for the zone, including Manhattan, Brooklyn, Queens, Bronx, Staten Island, EWR, Unknown, and dirty NaN labels. + tests: + - not_null: + severity: warn diff --git a/models/staging/stg_trips.sql b/models/staging/stg_trips.sql index b5f7b81..8f17a68 100644 --- a/models/staging/stg_trips.sql +++ b/models/staging/stg_trips.sql @@ -3,11 +3,13 @@ -- Downstream: fct_daily_borough_stats joins this to stg_zones. SELECT - -- TODO: select the columns you need for the mart: - -- pickup_datetime, pickup_location_id, fare_amount, tip_amount, trip_distance - -- - -- TODO: add tip_pct using {{ safe_divide('tip_amount', 'fare_amount') }} - -- - -- TODO: filter out rows where pickup_location_id IS NULL or fare_amount < 0 - + pickup_datetime, + pickup_location_id, + fare_amount, + tip_amount, + trip_distance, + {{ safe_divide('tip_amount', 'fare_amount') }} as tip_pct FROM {{ source('nyc_taxi', 'raw_trips') }} +where pickup_location_id is not null + and fare_amount >= 0 + diff --git a/models/staging/stg_zones.sql b/models/staging/stg_zones.sql index cc34d23..b378c7a 100644 --- a/models/staging/stg_zones.sql +++ b/models/staging/stg_zones.sql @@ -2,6 +2,6 @@ -- Exposes location_id and borough for use as a lookup in the mart. SELECT - -- TODO: select location_id and borough from {{ source('nyc_taxi', 'raw_zones') }} - + location_id, + borough FROM {{ source('nyc_taxi', 'raw_zones') }} diff --git a/package-lock.yml b/package-lock.yml new file mode 100644 index 0000000..f683b32 --- /dev/null +++ b/package-lock.yml @@ -0,0 +1,5 @@ +packages: + - name: dbt_utils + package: dbt-labs/dbt_utils + version: 1.4.1 +sha1_hash: e6424ba9e5a22487e47f023803aa4f0411946808 diff --git a/packages.yml b/packages.yml index bbb2357..fb5320a 100644 --- a/packages.yml +++ b/packages.yml @@ -1,5 +1,7 @@ -# TODO: Task 5 -- declare the dbt-labs/dbt_utils package here, then run + # `dbt deps` to install it. You need it for the compound uniqueness test # on the mart. See https://hub.getdbt.com/dbt-labs/dbt_utils/latest/ # for the package block syntax. -packages: [] +packages: + - package: dbt-labs/dbt_utils + version: [">=1.1.0", "<2.0.0"] diff --git a/profiles.yml.example b/profiles.yml.example index 82c6ae0..18dca60 100644 --- a/profiles.yml.example +++ b/profiles.yml.example @@ -3,14 +3,11 @@ nyc_taxi_borough_daily: outputs: dev: type: postgres - host: "{{ env_var('PG_HOST') }}" + host: hyf-data-pg.postgres.database.azure.com port: 5432 - user: "{{ env_var('PG_USER') }}" + user: hyfadmin password: "{{ env_var('PG_PASSWORD') }}" - dbname: "{{ env_var('PG_DBNAME', 'postgres') }}" - schema: "dev_" # TODO: replace with your first name (the schema you already own) - threads: 1 - -# Copy this file to profiles.yml (same directory), fill in your name, and ensure -# PG_HOST, PG_USER, PG_PASSWORD, and PG_DBNAME are set in your environment. -# profiles.yml is git-ignored — never commit it with a real password. + dbname: team1 + schema: dev_ + sslmode: require + threads: 4 diff --git a/reports/answers.md b/reports/answers.md index 8fb8cf2..e5df234 100644 --- a/reports/answers.md +++ b/reports/answers.md @@ -7,12 +7,22 @@ Queries run against `dev_.fct_daily_borough_stats`. **SQL:** ```sql --- TODO: query fct_daily_borough_stats grouped by pickup_borough, sum total_fare, order DESC +select + pickup_borough, + round(sum(total_fare)::numeric, 2) as dataset_total_fare_usd +from dev_halyna.fct_daily_borough_stats +group by pickup_borough +order by dataset_total_fare_usd desc +limit 1; ``` -**Result:** TODO +**Result:** -**Interpretation:** TODO (one sentence) +| pickup_borough | dataset_total_fare_usd | +| -------------- | ---------------------: | +| Manhattan | 493955.62 | + +**Interpretation:** Manhattan had the highest total fare in the loaded dataset, with total fare revenue of $493,955.62. --- @@ -21,12 +31,22 @@ Queries run against `dev_.fct_daily_borough_stats`. **SQL:** ```sql --- TODO: query fct_daily_borough_stats grouped by pickup_date, sum trip_count, order DESC LIMIT 1 +select + pickup_date, + sum(trip_count) as total_trips +from dev_halyna.fct_daily_borough_stats +group by pickup_date +order by total_trips desc +limit 1; ``` -**Result:** TODO +**Result:** + +| pickup_date | total_trips | +| ----------- | ----------: | +| 2024-01-17 | 2221 | -**Interpretation:** TODO (one sentence) +**Interpretation:** January 17, 2024 had the highest overall trip volume in the loaded dataset, with 2,221 trips across all pickup boroughs. --- @@ -35,12 +55,27 @@ Queries run against `dev_.fct_daily_borough_stats`. **SQL:** ```sql --- TODO: query fct_daily_borough_stats order by avg_tip_pct DESC LIMIT 5 +select + pickup_borough, + pickup_date, + trip_count, + round(avg_tip_pct::numeric, 4) as avg_tip_pct +from dev_halyna.fct_daily_borough_stats +order by avg_tip_pct desc nulls last +limit 5; ``` -**Result:** TODO +**Result:** -**Interpretation:** TODO — note whether any avg_tip_pct > 1 rows appear and what causes them +| pickup_borough | pickup_date | trip_count | avg_tip_pct | +| -------------- | ----------: | ---------: | ----------: | +| Unknown | 2024-01-30 | 4 | 2.5001 | +| Unknown | 2024-01-07 | 3 | 1.3396 | +| Unknown | 2024-01-11 | 8 | 1.1045 | +| Unknown | 2024-01-16 | 2 | 1.0000 | +| Unknown | 2024-01-18 | 3 | 0.6111 | + +**Interpretation:** The highest average tip percentage was 2.5001 for the Unknown borough on 2024-01-30. This is an outlier from a very small group of only 4 trips, so it should be treated as a data-quality or small-sample warning rather than a typical tipping pattern. --- @@ -49,9 +84,20 @@ Queries run against `dev_.fct_daily_borough_stats`. **SQL:** ```sql --- TODO: use percentile_cont(0.5) WITHIN GROUP (ORDER BY trip_count) filtered by borough +select + pickup_borough, + percentile_cont(0.5) within group (order by trip_count) as median_daily_trip_count +from dev_halyna.fct_daily_borough_stats +where pickup_borough in ('Manhattan', 'Brooklyn') +group by pickup_borough +order by pickup_borough; ``` -**Result:** TODO +**Result:** + +| pickup_borough | median_daily_trip_count | +|---|---:| +| Brooklyn | 248 | +| Manhattan | 1169.5 | -**Interpretation:** TODO (one sentence on the ratio) +**Interpretation:** Manhattan had a much higher median daily trip volume than Brooklyn: 1,169.5 trips per day compared with 248 trips per day. The `.5` value appears because `percentile_cont` calculates a continuous median and can interpolate between two middle daily counts. diff --git a/tests/assert_avg_tip_pct_within_bounds.sql b/tests/assert_avg_tip_pct_within_bounds.sql index b0fab32..5e94f58 100644 --- a/tests/assert_avg_tip_pct_within_bounds.sql +++ b/tests/assert_avg_tip_pct_within_bounds.sql @@ -13,7 +13,8 @@ -- -- The test passes (no WARN) when zero rows are returned; any returned rows are flagged. --- TODO: write the SELECT here. -- Query {{ ref('fct_daily_borough_stats') }} and return rows where avg_tip_pct > 1. -SELECT NULL AS pickup_borough, NULL AS pickup_date, NULL AS avg_tip_pct -WHERE FALSE -- TODO: replace with the real query +SELECT pickup_borough, pickup_date, avg_tip_pct +FROM {{ ref('fct_daily_borough_stats') }} +WHERE avg_tip_pct > 1 +