Adding SIGTERM handling for graceful MD exit

[sihoonchoi]

Summary of Changes

Following up on #3456. Added SIGTERM handling to guarantee a graceful MD exit before the walltime limit, providing a fallback when max_runtime is not declared.

Requirements

• My PR is focused on a single feature addition or bugfix.
• My PR has relevant, comprehensive unit tests.
• My PR is on a custom branch (i.e. is not named main).

Note: If you are an external contributor, you will see a comment from @buildbot-princeton. This is solely for the maintainers.

[buildbot-princeton]

Can one of the admins verify this patch?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (44db3f3) to head (0d95626).

Additional details and impacted files

@@            Coverage Diff            @@
##              main     #3465   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          101       101           
  Lines         4375      4399   +24     
=========================================
+ Hits          4375      4399   +24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sihoonchoi]

@Andrew-S-Rosen What I’ve been trying to do here is catch a SIGTERM signal from the scheduler so that the MD run can be terminated safely. However, it seems that if the signal is sent while a single dyn.irun is running, the defined trigger function is not called, so the MD run continues until the walltime is reached. This happens even with a very simple MD setup, where a single dyn.irun takes only a fraction of a second. Therefore, I think the previous setup (#3456) handles this issue better.

The other option would be to get the walltime from the batch submission file, but it looks like accessing that information through os.environ.get is restricted (at least on Princeton Research Computing). Therefore, explicitly defining max_runtime seems to be the best way to ensure that the MD run exits gracefully before reaching the walltime.