Yet another alternative implementation for working with derived variables

[bouweandela]

Description

Yet another alternative implementation to #2777 and #3051 for finding input datasets for derived variables and loading them. Adds a new class method `esmvalcore.dataset.DerivedDataset that can be used to work with derived variables.

Note that derived variables with optional input datasets are not yet supported. The only derived variable that uses this feature is amoc at the moment, so the impact of that is limited.

Link to documentation: https://esmvaltool--3053.org.readthedocs.build/projects/ESMValCore/en/3053/notebooks/derived-variables.html

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

❌ Patch coverage is 37.50000% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.00%. Comparing base (5ca0b1d) to head (de9351d).

Files with missing lines	Patch %	Lines
esmvalcore/dataset.py	37.50%	25 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3053      +/-   ##
==========================================
- Coverage   96.15%   96.00%   -0.15%     
==========================================
  Files         270      270              
  Lines       15805    15844      +39     
==========================================
+ Hits        15197    15211      +14     
- Misses        608      633      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[schlunma]

Thanks Bouwe, looks good! I prefer this approach over #3051. This will certainly be very helpful when using the ESMValCore Python API.

I am not a very big fan of adding a feature to the Dataset class but not using it when parsing recipes, but in this case it's much easier to separate this. Incorporating this cleanly into the existing code was by far the hardest problem I had in #3051.

Will look into this in more detail once the PR is ready for review! Thanks so much!

[schlunma]

Looks like this is not used anywhere

[schlunma]

Suggested change

	def from_files(self) -> Iterator[Self]:
	def from_files(self) -> Generator[Self]:

I think a Generator is more accurate (every Generator is an Iterator, but not vice versa).