Skip to content

Open World Patient Ranking Pipeline#286

Open
eveenhuis wants to merge 14 commits into
mainfrom
dev/rebase-open-world-ranked
Open

Open World Patient Ranking Pipeline#286
eveenhuis wants to merge 14 commits into
mainfrom
dev/rebase-open-world-ranked

Conversation

@eveenhuis

@eveenhuis eveenhuis commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Rebased version of !283, includes changes from !273

See the description from those below:

  • Fixes unit tests broken by merit focus weight modification from ADEPT 2026-01-21
  • We weren't finding that medical urgency/KDMA(s) varied across actions for the same character.So, we are now regressing based on character descriptions only (or non-character actions, but that doesn't really apply due to the available actions). [OWFormatChoicesADMComponent, then CR/DR]
  • We use alignment to pick the highest ranked character (or non-character action) [Existing alignment function].
  • That character may correspond to multiple actions, so we then have a secondary non-aligned prompt to choose which sub action to do. [OWChoiceToActionADMComponent]
  • Once we've picked a single action, we may have details to fill in like tag color or character ID, so we then do that with followups (again, not aligned) [OWActionParameterCompletionADMComponent]
  • Then finally, we do a tag adjustment based on a heuristic to "align" tag color. [OWTaggingAdjustmentADMComponent]. I didn't want to name this component alignment because it doesn't directly use the alignment target. Instead it considers the KDMA ranking vs medical only ranking as well as how other patients have been tagged so far

@eveenhuis
eveenhuis changed the base branch from main to dev/rebase-open-world-ph2 July 23, 2026 21:08
@eveenhuis
eveenhuis changed the base branch from dev/rebase-open-world-ph2 to main July 23, 2026 21:08
@eveenhuis eveenhuis changed the title Dev/rebase open world ranked Open World Patient Ranking Pipeline Jul 23, 2026
@eveenhuis
eveenhuis requested a review from dmjoy July 23, 2026 21:18
@eveenhuis
eveenhuis marked this pull request as ready for review July 23, 2026 21:19

@dmjoy dmjoy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly minor comments and overall LGTM. Perhaps most pressing would be to add a bit more to the Changelog, but the other things can be deferred to another PR (or included in a subsequent PR for OW).

Comment on lines +104 to +105
self.followup_character_prompt = FollowupClarifyCharacterPrompt()
self.followup_tag_prompt = FollowupClarifyTagPrompt()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should probably be parameterized (i.e. added to the __init__ args)

Comment on lines +2 to +10
defaults:
- /adm_component/misc@adm.step_definitions.action_parameter_completion: ow_action_parameter_completion
- /adm_component/misc@adm.step_definitions.tagging_adjustment: ow_tagging_adjustment
- override /adm: phase2_pipeline_direct_regression
- override /interface: ta3
- override /adm_component/misc@adm.step_definitions.format_choices: ow_format_choices
- override /adm_component/alignment@adm.step_definitions.scalar_alignment: tournament_random_effects_tuple
- override /adm_component/misc@adm.step_definitions.ensure_chosen_action: ow_choice_to_action
- override /driver: itm_phase2_ow

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if it makes more sense to have new adm configs (I guess we would just need one for comp reg and one for direct reg) to base these experiment files off of.

Comment thread CHANGELOG.md

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect there's a few more things we'd want to add to the changelog (new components etc.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants