t.list: Add support for listing multiple dataset types - #7731
Conversation
ninsbl
left a comment
There was a problem hiding this comment.
Looks in general good to me. I have only minor comments inline below...
|
@petrasovaa it would be good if you could give your opinion on how JSON output should be handled / look like. Where should mapset or type go for example and should the json structure be consistent for single an multiple requested types I probably thonk yes... |
So, to avoid different sets of keys in each item in JSON (raster vs strsds), how about allowing user to just combine the same type (e.g. strds, stvds) and not allow combining e.g. raster and strds. I would like to keep the flat structure of the JSON. |
ninsbl
left a comment
There was a problem hiding this comment.
I tried to think through the PR on a bus once more. Sorry for bringing up some new stuff. I have no strong opinion there, but I would be glad if you could consider my comments and address them as you see fit.
| for dtype in type: | ||
| for mapset in dbif.tgis_mapsets: | ||
| if temporal_type == "absolute": | ||
| table = dtype + "_view_abs_time" |
There was a problem hiding this comment.
If the intention is to be able to list all STDS in one go, should we not support also multiple / list input for temporal_type?
Sorry for bringing that up somewhat late...
There was a problem hiding this comment.
I agree and t.list already allows that, just not the function.
There was a problem hiding this comment.
That probably means the temporal_type should always go into the JSON and CSV output too, no?
There was a problem hiding this comment.
The temporal_type is already in CSV and JSON with columns=all, it could be added explicitly as one of the options, but I am not sure it's worth it.
| for row in rows: | ||
| row_dict = dict(row) | ||
| if len(type) > 1: | ||
| row_dict["type"] = dtype |
There was a problem hiding this comment.
Back to the question if the keys should be consistent, regardless if t.list is run with multiple or single type (or possibly single or multiple temporal_type input? I tend to say: make the dict (JSON) output structure consistent / predictable. But no strong opinion. It should just be deliberate.
Also, could this block be solved simpler with a dict-comprehension and/or a dict-update? (consult ruff --preview --select ALL)
There was a problem hiding this comment.
The JSON should not have mixed items (each item in an array should have the same set of keys), which is the case now because we don't mix maps and datasets. In the comment above I was thinking to add the type key always for JSON and CSV. I hope I am not missing anything here.
There was a problem hiding this comment.
Right, but e.g semantic_label is only available for STRDS. So, should columns that do not exist for every dataset type just be empty? Just me thinking out loud...
There was a problem hiding this comment.
I commented on this below, I think we need to select the intersection of the different columns.
|
I've tried to cover all the suggested changes:
I hope I haven't missed anything! Also, apologies for the late response, I was a bit occupied with other things. |
ninsbl
left a comment
There was a problem hiding this comment.
In other, similar tools, JSON output defaults to all columns in the output, if I remember correctly?!?
How is this handled here? Different types can have different columns...
Yes, In Here is an example output of [
{
"id": "nc_lst_daily@PERMANENT",
"name": "nc_lst_daily",
"mapset": "PERMANENT",
"temporal_type": "absolute",
"semantic_type": "mean",
"creation_time": "2026-06-03 18:15:06",
"modification_time": "2026-06-03 18:18:05",
"creator": "saket",
"start_time": "2002-01-01 00:00:00",
"end_time": "2004-10-05 00:00:00",
"granularity": "1 day",
"north": 279073.975466,
"south": 113673.975466,
"east": 798143.311797,
"west": 595143.311797,
"bottom": 0.0,
"top": 0.0,
"proj": "XY",
"raster_register": "raster_map_register_9ca4f56710e5453291c0c830aa8f9a92",
"number_of_maps": 1008,
"nsres_min": 200.0,
"ewres_min": 200.0,
"nsres_max": 200.0,
"ewres_max": 200.0,
"min_min": -1.49,
"min_max": 30.63,
"max_min": -0.29,
"max_max": 48.23,
"aggregation_type": null,
"title": "NC Daily LST",
"description": "Standard MODIS LST Time Series for North Carolina",
"command": "...",
"type": "strds"
},
{
"id": "schools_stds@PERMANENT",
"name": "schools_stds",
"mapset": "PERMANENT",
"temporal_type": "absolute",
"semantic_type": "mean",
"creation_time": "2026-06-10 17:50:37",
"modification_time": "2026-06-10 17:52:46",
"creator": "saket",
"start_time": "2024-01-01 00:00:00",
"end_time": "2027-01-01 00:00:00",
"granularity": "1 year",
"north": 248159.844411,
"south": 203559.011362,
"east": 671714.551108,
"west": 619215.133883,
"bottom": 0.0,
"top": 0.0,
"proj": "XY",
"vector_register": "vector_map_register_e0d7ff31f8dc41de82128f9a9c1e237b",
"number_of_maps": 3,
"title": "Wake County Schools",
"description": "Test dataset for GSoC",
"command": "...",
"points": 501,
"lines": 0,
"boundaries": 0,
"centroids": 0,
"faces": 0,
"kernels": 0,
"primitives": 501,
"nodes": 0,
"areas": 0,
"islands": 0,
"holes": 0,
"volumes": 0,
"type": "stvds"
}
] |
|
Some issues found (they might be pre-existing):
|
|
Regarding the different keys for the different types, that is indeed a problem I didn't realize. Currently the CSV with mixed types won't work at all, the header won't match. So unfortunately this is getting more complicated with the "columns=all" option. I think we need to select only keys that intersect for the different groups. |
|
I was going back to the "type" option we had previously as one of the options in the columns option. I am now thinking this might be a better option. So if user doesn't request it (e.g. columns=id) it won't show up (which might be more expected). Now when user requests multiple types and columns is not explicitly specified, the type would show up in JSON (default is columns=all behavior) and in CSV (the default is columns=id,type). When single type is requested, I am now leaning to omit it (unless explicitly requested with columns option). Sorry for keep changing the design, this has been somewhat more complicated than I hoped... |
|
I also realized the output of the function changed, basically breaking the API. Could we return the previous output if single type was selected? So if type is a str, return exactly the old shape (raw row objects, no injected key), if it's a list, return the new dict form. |
This PR implements the feature requested in #7695 to support listing multiple dataset types simultaneously, along with improvements in t.list tests.
@ninsbl @petrasovaa