Develop best practices for handling non-spatiotemporal dimensions

[chris-little]

This issue is to discuss how to support custom extra dimensions, such as "wavelength". The custom dimensions may not even be ordered or implictly continuous, like (x, y, z, t). A collection of ensembles of forecast data is one example of a discrete and unordered dimension. (but see Issue #172 for support of ensembles specifically)

[jonblower]

Just to bring the key info over from the previous discussion - CovJSON already supports arbitrary numbers of dimensions. The use of domainTypes can restrict this to reduce the multiplicity of options that clients have to deal with. So there isn't a problem with the schema (as far as I know).

So I think this is more a case of working out what is "best practice" in different situations. Is it best to define new domainTypes? Or model the dimensions as variables (e.g. in a multispectral image, the different wavelengths could be modelled as dimensions or variables).

I'd suggest renaming this issue "Develop best practices for handling non-spatiotemporal dimensions" or something like that.

[havardf]

Has there been any more work done on this? Or maybe some live examples for how to represent this? We are looking to solve this problem now.

[jonblower]

Hi @havardf, what is your use case? The spec already allows for non-spatiotemporal dimensions, so you may be able to achieve what you want already. The question would be whether there is a case for new domain types or vocabulary terms that might aid interoperability

[havardf]

@jonblower Thank you for the swift response! The use-case is pretty much to get a forecast timeseries or a vertical profile for a position from an ensemble forecast. So, its about how to represent that result with CoverageJSON.

And I think its not so much figuring out how to do technically, as to figure out the best way to do it so that multiple NMHSs can agree on the same way of doing it. And getting consensus might be easier if there are pre-existing examples / best practices that we can refer to.

[jonblower]

@havardf thanks for this - unfortunately I don't think there is much precedent for this. There is a discussion in #172 about ensembles which discusses the issues and tentatively proposes some approaches, but no we didn't get to a consensus.

Broadly, I think the options were:

1. Use a dimension to represent the ensemble. Disadvantage is that it fails schema checks if you want to use the existing domain types. The domain types could be extended to accommodate ensembles, but that would require discussion.
2. Represent each ensemble member as a separate Coverage, and group them in a CoverageCollection. There may be some redundancy of metadata, but this works "out of the box".
3. There was a more tentative suggestion in How best to support Ensembles of data #172 to encode the ensemble members using the Range, which would need to be explored. This would probably only make sense for ensembles, not general custom dimensions.

Do you have a preference? The key advantage of option 1 is the that you can slice on the ensemble dimension ("give me the values at this point from all ensemble members"), although that's not too hard to do with option 2 in practice.

[havardf]

Sorry for the late reply:

No, not really any preference currently, but I will try to get more people engaged on this. I guess we just have to try this to get a feel for which option works best in real use-cases.

[jonblower]

Thanks @havardf. For what it's worth, my personal preference is Option 2, but these days I am not the person who has to implement this! I think it depends on what you are likely to want to do with an ensemble.

[mrauhala]

I was also thinking about this, and ended up with option 2. But I don't think it is good enough, as clients would have to specifically know about this to fully use ensemble data. But I might work in some form out of the box for some clients.

What I came up with

CoverageCollection (domainType: "PointSeries")
│
├── metocean:ensemble          ← extension: ensemble metadata
│   ├── totalMembers: 40
│   └── idPattern: "metocean:member-{index}"
│
├── parameters (shared)        ← defined once, inherited by all coverages
│   ├── air_temperature        ← keyed by CF standard name
│   │   ├── observedProperty   → NERC vocab URI
│   │   └── unit.symbol        → WMO codes.wmo.int URI
│   └── wind_speed
│       ├── observedProperty   → NERC vocab URI
│       └── unit.symbol        → WMO codes.wmo.int URI
│
├── referencing (shared)       ← defined once, inherited by all coverages
│   ├── [x, y] → CRS84
│   └── [t]    → Gregorian
│
└── coverages[]                ← one per ensemble member
    ├── Coverage (id: "metocean:member-0")
    │   ├── domain: { x, y, t }     ← standard PointSeries axes
    │   └── ranges:
    │       ├── air_temperature → NdArray [t]
    │       └── wind_speed      → NdArray [t]
    ├── Coverage (id: "metocean:member-1")
    │   └── ...
    └── Coverage (id: "metocean:member-39")
        └── ...

[mrauhala]

Above solution is otherwise good as it works for multiple DomainTypes, but you would have to pick DomainType based on query shape for example MULTIPOINT queries, you would have to return MultiPointSeries.

CoverageCollection (domainType: "MultiPointSeries")
└── coverages[]
    ├── Coverage (id: "metocean:member-0")   ← still unique
    │   └── domain:
    │       ├── composite: [[24.94, 60.17], [25.50, 61.00]]  ← all points
    │       └── t: [6 timesteps]
    │   └── ranges:
    │       └── air_temperature → NdArray [composite, t] shape [2, 6]
    │
    ├── Coverage (id: "metocean:member-1")   ← still unique
    │   └── ...

But I still don't like this, It would be better to be able to allow custom axes. Even new DomainType does not necessarily solve this, as for different queries you would need to return grid, pointseries, verticalprofile and so on..

[jonblower]

I like the idea of a simple extension to add a "metocean:ensemble" vocabulary term (or something like it) to denote that a CoverageCollection is an ensemble.

I didn't quite understand the point in your second post (immediately above this one). Are you trying to handle the use case of querying multiple ensemble members at the same time (and hence getting an ensemble of results)?