Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions OLIDS/Documentation/Schema/Appointment.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,9 @@ This definition takes the concepts of appointments in a clinical setting and als
| `PLANNED_DURATION_MINS` | `NUMBER` | planned duration. | | | `planned_duration` |
| `ACTUAL_DURATION_MINS` | `NUMBER` | actual duration. | | | `actual_duration` |
| `APPOINTMENT_STATUS_SOURCE_CONCEPT_ID` | `UUID` | appointment status concept id. | FK -> [CONCEPT](concept.md).ID| | `appointment_status_concept_id` |
| `SLOT_STATUS_RAW` | `VARCHAR` | appointment slot status description as supplied in source dataset | | ❌ Column removed | |
| `APPOINTMENT_STATUS_DDS` | `VARCHAR` | calculated appointment status from logic | | ❌ Column removed | |
| `SLOT_STATUS_RAW` | `VARCHAR` | (⚠️ **deprecated**) appointment slot status description as supplied in source dataset | | ❌ Column removed | |
| `APPOINTMENT_STATUS_DDS` | `VARCHAR` | (⚠️ **deprecated**) calculated appointment status from logic | | ❌ Column removed | |
| `APPOINTMENT_STATUS_DERIVED` | `VARCHAR` | calculated appointment status from logic | | | |
| `PATIENT_WAIT_MINS` | `NUMBER` | patient wait. | | | `patient_wait` |
| `PATIENT_DELAY_MINS` | `NUMBER` | patient delay. | | | `patient_delay` |
| `DATETIME_BOOKED` | `TIMESTAMP_NTZ` | date time booked. | | | -- |
Expand Down
26 changes: 26 additions & 0 deletions OLIDS/Documentation/Schema/Episode_Of_Care.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
- [Columns](#columns)
- [Entity relationships](#entity-relationships)
- [Notes](#notes)
- [Episode of Care History](#episode-of-care-history)
- [Questions on Episode of Care History](#questions-on-episode-of-care-history)
- [Does this change affect historic clinical events?](#does-this-change-affect-historic-clinical-events)
- [How does this change affect population count comparissons?](#how-does-this-change-affect-population-count-comparissons)

## Overview

Expand Down Expand Up @@ -79,3 +83,25 @@ erDiagram
| [Practitioner_In_Role](Practitioner_In_Role.md) | FK | USUAL_GP_PRACTITIONER_IN_ROLE_ID | ID | |

## Notes

### Episode of Care History

As of [v3.0.1](../../Release-notes/3_OLIDS_Enrichment/v3.0.1.md), the episode of care (`EPISODE_OF_CARE_V2`) table incorporates historic registration data available from the EMIS source object `admin_patient_history`.

The inclusion of this data allows us to close a gap whereby patients who left and rejoined a practice one or more times prior to the first bulk ever taken from the practice would only have their most recent registration period noted within the Episode of Care table. This led to a small variance in the number of registered patients when looking for registered population counts prior to the first bulk, a variance which grew in size the further back prior to the first bulk you looked.

The fix will close the gap up to the maximum limit of five (5) years prior to the first bulk date, as patients who were deducted more than five (5) years prior to the first bulk are not possible to include in the EMIS IM1 extracts.

Users will still see a variance between referenced population when counting the total population as of a date that is more than 5 years prior to the first bulk (typically around 2020/2021).

#### Questions on Episode of Care History

##### Does this change affect historic clinical events?

No. this change only impacts the count of episodes of care.

Clinical event data such as medications, observations, referrals, and allergies are unaffected by this change. These will continue to hold all records for patients who are currently registered, or were registered within the preceeding five (5) years prior to the first bulk date of the practice.

##### How does this change affect population count comparissons?

This change will increase the number of registered regular patients for periods that are up to five (5) years prior to the date of the first bulk from the practice
48 changes: 48 additions & 0 deletions OLIDS/Documentation/Schema/Location_Contact.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Location Contact

- [Location Contact](#location-contact)
- [Overview](#overview)
- [Columns](#columns)
- [Entity Relationships](#entity-relationships)
- [Notes](#notes)

## Overview

Linked FHIR resource: [🔥 Location](https://hl7.org/fhir/location.html)

Details and position information for a place where services are provided and resources and participants may be stored, found, contained, or accommodated.

A Location includes both incidental locations (a place which is used for healthcare without prior designation or authorization) and dedicated, formally appointed locations. Locations may be private, public, mobile or fixed and scale from small freezers to full hospital buildings or parking garages.

## Columns

| Column Name | Data Type (Size) | Description | PK/FK | Compass Equivalent |
| --- | --- | --- | --- | --- |
| `ID` | `UUID` | id. | PK | `id` |
| `LDS_SOURCE_RECORD_ID` | `UUID` | Unique record identifier including file row number for deduplication. | | -- |
| `LOCATION_ID` | `UUID` | the id of the linked location for this contact | FK -> [`LOCATION`](Location.md).`ID` | |
| `IS_PRIMARY_CONTACT` | `BOOLEAN` | True where the contact is noted as the primary contact for the location. False where not. Not all locations will denote a primary contact. | | |
| `CONTACT_TYPE_SOURCE_CONCEPT_ID` | `UUID` | The Concept ID for the type of contact (phone/email etc) | FK -> [`CONCEPT`](Concept.md).`ID` | |
| `CONTACT_TYPE` | `VARCHAR` | The type of contact (phone / email etc) | | |
| `VALUE` | `VARCHAR` | The contact number/information | | |
| `SOURCE_EXTRACTION_DATE` | `TIMESTAMP_NTZ` | source extraction date. | | -- |
| `LDS_SOURCE_DATASET` | `VARCHAR` | The name of the source dataset (or system) that the record is obtained from | - | -- |
| `LDS_TRANSFORM_DATETIME` | `TIMESTAMP_LTZ` | lds transform date time. | | -- |

## Entity Relationships

> [!NOTE]
> Diagrams below are currently indicative. The precise optional/mandatory nature of certain relationships remains to be clarified.

```mermaid
erDiagram
LOCATION ||--o{ LOCATION_CONTACT: location_id
CONCEPT ||--o{ LOCATION_CONTACT: contact_type_source_concept_id
```

| Related Table | Relationship Type | Local Key | Related Key | Notes |
| --- | --- | --- | --- | --- |
| [Location](Location.md) | FK | LOCATION_ID | ID | |
| [Concept](Concept.md) | FK | CONTACT_TYPE_SOURCE_CONCEPT_ID | CONCEPT_ID | |

## Notes
31 changes: 31 additions & 0 deletions OLIDS/Release-notes/1_OLIDS_Share/v1.5.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Release v1.5.0 - 2026-08-13

> [!NOTE]
> The below is a summary of the changes since the previous release [v1.4.0](v1.4.0.md).

## Summary

This release includes an update to the `duckdb__format_date_to_string` function, enhancing its ability to handle postcode version date strings in the `'AUG2026'` format. This improvement increases compatibility with datasets that use this specific date representation, ensuring smoother data processing and formatting.

### Improvements

- Updated `duckdb__format_date_to_string` to support postcode version date string format (e.g., `'AUG2026'`). [PR: #18](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/18)
> 🎯 ***Impact:*** Enables correct formatting and parsing of postcode version dates, improving data compatibility and reducing potential errors when handling date strings in this format.

### Inputs

This changelog was automatically produced from:

- 1 Pull Requests
- 0 Issues
- 0 WorkItems
- 2 Commits

## Full Details

Pull Requests included in this release:

- [PR #18](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/18): Updated duckdb__format_date_to_string to handle postcode version date…
- No linked issues.

[View the diff between v1.4.0 and v1.5.0](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/compare/releases%2Fv1.4.0...releases%2Fv1.5.0)
61 changes: 61 additions & 0 deletions OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Release v2.5.0 - 2026-08-17

> [!NOTE]
> The below is a summary of the changes since the previous release [v2.3.0](v2.3.0.md).

## Summary

This release introduces versioning to the `PC_EMIS_Episode_Of_Care` transformed model, enabling support for historic registrations and improved data lineage, alongside a new derived appointment status column on the `PC_EMIS_Appointment` model. Referral request logic in medication models has been streamlined to remove ambiguous references to `Observation_type=referral`. Documentation has been enhanced with explicit version tracking and column history, unit test coverage has been refined, and the OLIDS_Share dependency has been updated to revision v1.5.0. Several internal improvements and bug fixes further strengthen data quality and maintainability.

### New Features

- Added version 2 of the `PC_EMIS_Episode_Of_Care` transform to support historic registrations by consuming data from the `ADMIN_PATIENT_HISTORY` table. Introduced model versioning semantics and updated documentation to reflect changes. *[PR: [#116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116)]*
> ✨ ***Feature:*** Users can now access historic registration data, improving the completeness and traceability of episode-of-care records.

- Add derived column for appointment status and update tests *[PR: [#117](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/117)]*
> ✨ ***Feature:*** Introduces a new `APPOINTMENT_STATUS_DERIVED` column in the appointments model, preparing for future schema changes and improving clarity in appointment status handling.

### Improvements

- Enhanced the `PC_EMIS_Episode_Of_Care` model documentation with explicit version tracking and column history. Updated unit tests to ensure they target the correct model version for reliable validation as the model evolves. Added the `PATIENT_TYPE_DESCRIPTION` field to the sequenced admin patient history model and SQL transformation. *[PR: [#116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116)]*
> 🎯 ***Impact:*** Improves transparency, test coverage, and data clarity for end users and maintainers.

- Update OLIDS_Share to revision v1.5.0 *[PR: [#119](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/119)]*
> 🎯 ***Impact:*** Ensures the project uses the latest OLIDS_Share features and bug fixes, supporting ongoing compatibility and stability.

### Bug Fixes

- Corrected macro enclosure invocation for `generate_sha256` and resolved issues in model scenarios identified by tests. *[PR: [#116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116)]*
> 🐞 ***Fix:*** Ensures accurate hash generation and robust model behavior, reducing the risk of data inconsistencies.

- Remove remaining references to Observation_type=referral *[PR: [#118](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/118)]*
> 🐞 ***Fix:*** Updates referral request logic in medication models to rely solely on `EMIS_CODE_CATEGORY_DESCRIPTION = 'Referral'`, eliminating ambiguous use of `OBSERVATION_TYPE` and improving data accuracy.

### Refactoring

- Refactored the `PC_EMIS_Episode_Of_Care` transform by separating state change detection into its own CTE and improving SQL logic clarity. Removed the unversioned model to streamline codebase consistency. *[PR: [#116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116)]*
> ♻️ ***Refactor:*** Enhances maintainability and readability, making future updates and debugging more efficient.

### Inputs

This changelog was automatically produced from:

- 4 Pull Requests
- 0 Issues
- 0 WorkItems
- 14 Commits

## Full Details

Pull Requests included in this release:

- [PR #116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116): add PC_EMIS_Episode_Of_Care (version 2) - historic registrations
- No linked issues.
- [PR #117](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/117): Add derived column for appointment status and update tests
- No linked issues.
- [PR #118](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/118): Remove remaining references to Observation_type=referral
- No linked issues.
- [PR #119](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/119): Update OLIDS_Share to revision v1.5.0
- No linked issues.

[View the diff between v2.3.0 and v2.5.0](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/compare/releases%2Fv2.3.0...releases%2Fv2.5.0)
43 changes: 43 additions & 0 deletions OLIDS/Release-notes/3_OLIDS_Enrichment/v2.3.1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Release v2.3.1 - 2026-08-06

> [!NOTE]
> The below is a summary of the changes since the previous release [v2.3.0](v2.3.0.md).

## Summary

This release focuses on improving data governance, security, and maintainability across the enrichment pipeline. Key changes include streamlining the UPRN schema by removing obsolete columns, enhancing the application of masking and row-level security policies, and refactoring configurations to leverage dbt-Snowflake features more effectively. These updates strengthen data protection, ensure schema consistency, and simplify ongoing maintenance.

### Improvements

- Remove UPRN columns no longer in response, update stored procedures, and align all related models and tests to the new schema. *[PR: [#65](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/65)]*
> 🎯 ***Impact:*** Streamlines the UPRN enrichment pipeline by eliminating unused columns, ensuring consistent schema definitions, and improving data clarity and maintainability.

- Add post-hook for PDS model to apply masking policies. *[PR: [#66](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/66)]*
> 🎯 ***Impact:*** Automates the application of masking policies and access permissions on deployment, enhancing data protection and reducing manual configuration errors.

### Refactoring

- Refactor `Person_PDS_Latest.sql` to use dbt-Snowflake native row access policy configuration. *[PR: [#64](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/64)]*
> ♻️ ***Refactor:*** Moves row-level security configuration into the model definition for atomic application, improving security posture and simplifying future configuration management.

### Inputs

This changelog was automatically produced from:

- 3 Pull Requests
- 0 Issues
- 0 WorkItems
- 11 Commits

## Full Details

Pull Requests included in this release:

- [PR #64](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/64): Refactor `Person_PDS_Latest.sql` to use dbt-Snowflake native rap configuration
- No linked issues.
- [PR #65](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/65): Remove UPRN columns no longer in response
- No linked issues.
- [PR #66](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/66): Add post-hook for PDS model to apply masking policies
- No linked issues.

[View the diff between v2.3.0 and v2.3.1](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/compare/releases%2Fv2.3.0...releases%2Fv2.3.1)
36 changes: 36 additions & 0 deletions OLIDS/Release-notes/3_OLIDS_Enrichment/v2.3.2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Release v2.3.2 - 2026-06-08

> [!NOTE]
> The below is a summary of the changes since the previous release [v2.3.1](v2.3.1.md).

## Summary

This release focuses on improving the reliability and consistency of the NDOO and UPRN export processes. Key changes include enhanced candidate selection and deduplication logic for NDOO requests, improved handling of null values, and updates to ensure correct table and field usage in both NDOO and UPRN export procedures. These refinements help prevent duplicate requests, ensure accurate data exports, and maintain naming consistency across the platform.

### Improvements

- Remove unnecessary check for existing NHS_NUMBER in NDOO request processing, update candidate selection and deduplication logic, improve handling of null values, and ensure consistent table and column references in export procedures. *[PR: [#67](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/67)]*
> 🎯 ***Impact:*** Enhances accuracy in identifying candidates for NDOO requests, prevents duplicate or missed requests, and ensures export procedures operate on the correct data, improving overall data quality and process reliability.

- UPRN procedure change to pick up renamed export table and use the correct address field in batch exports. *[PR: [#69](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/69)]*
> 🎯 ***Impact:*** Ensures the UPRN export process references the correct table and address field, aligning with updated naming conventions and improving the consistency and accuracy of exported address data.

### Inputs

This changelog was automatically produced from:

- 2 Pull Requests
- 0 Issues
- 0 WorkItems
- 11 Commits

## Full Details

Pull Requests included in this release:

- [PR #67](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/67): Remove unnecessary check for existing NHS_NUMBER in NDOO request processing
- No linked issues.
- [PR #69](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/69): UPRN procedure change to pick up renamed export table
- No linked issues.

[View the diff between v2.3.1 and v2.3.2](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/compare/releases%2Fv2.3.1...releases%2Fv2.3.2)
Loading
Loading