HDXDSYS-2904 Add create_datastore and update_datastore methods to Resource in HDX Python API

.pre-commit-config.yaml

@@ -1,5 +1,5 @@
 default_language_version:
-  python: python3.13
+  python: python3
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks

CLAUDE.md

@@ -0,0 +1,57 @@
+# CLAUDE.md
+
+## Project Overview
+
+**hdx-python-api** is the official Python library for interacting with the [Humanitarian Data Exchange (HDX)](https://data.humdata.org/) platform. It provides ORM-like classes wrapping the CKAN API to read, create, update, and delete datasets, resources, organizations, users, vocabularies, showcases, and datastores.
+
+## Source Layout
+
+- `src/hdx/api/` — Configuration, session management, locations, utilities
+  - `configuration.py` — `Configuration` class (credentials, site URL, user agent)
+  - `remotehdx.py` — Low-level CKAN API client
+- `src/hdx/data/` — Data model objects, all inheriting from `HDXObject`
+  - `dataset.py`, `resource.py`, `organization.py`, `user.py`, `vocabulary.py`, `showcase.py`, `resource_view.py`
+  - `hdxobject.py` — Base class providing `_read_from_hdx` / `_write_to_hdx` and CRUD scaffolding
+- `src/hdx/facades/` — High-level entry-point helpers (`simple`, `keyword_arguments`, `infer_arguments`)
+- `tests/hdx/` — Test suite mirroring src layout; fixtures in `tests/fixtures/`
+
+## Running Tests
+
+```bash
+uv run pytest
+```
+
+Tests mock the CKAN HTTP session via inner `MockSession` classes in each test file — no live HDX connection needed for the standard suite. Integration tests require `HDX_KEY_TEST` and `HDX_PIPELINE_GSHEET_AUTH` environment variables.
+
+Coverage is written to `coverage.lcov` and JUnit XML to `test-results.xml`.
+
+## Code Style
+
+Formatted and linted with `ruff` (rules: E, F, I, UP; line-length not enforced). Run before committing:
+
+```bash
+pre-commit run --all-files
+```
+
+- Python ≥ 3.10
+- Type hints throughout; use `X | Y` union syntax (PEP 604), not `Optional`/`Union`
+- Google-style docstrings with `Args:` and `Returns:` sections
+- No inline comments unless the *why* is non-obvious
+
+## Key Patterns
+
+**HDXObject subclasses** expose a standard interface: `create_in_hdx()`, `update_in_hdx()`, `delete_from_hdx()`, `read_from_hdx()`. Each class defines an `actions()` static method mapping action names to CKAN API action strings.
+
+**Writing to HDX** uses `self._write_to_hdx(action_key, data_dict, id_field_name)` inherited from `HDXObject`. The `action_key` must be in `actions()`.
+
+**Test mocking** — each test fixture defines a `MockSession` with a `post()` method that inspects the URL and decoded JSON body to return `MockResponse` objects. State is tracked on class variables (e.g. `TestResource.datastore`).
+
+## Collaboration Style
+
+- Be objective, not agreeable. Act as a partner, not a sycophant. Push back when you disagree, flag tradeoffs honestly, and don't sugarcoat problems.
+- Keep explanations brief and to the point.
+- Don't rely on recalled knowledge for facts that could be stale (API behaviour, library versions, external systems). Search or read the actual source first. If you lack verified information, say so rather than speculate.
+
+## Scope of Changes
+
+When fixing a bug or addressing PR feedback, change only what is necessary to resolve the specific issue. Do not refactor surrounding code, rename variables, adjust formatting, or make improvements in the same commit unless they are directly required by the fix.

pyproject.toml

@@ -36,7 +36,7 @@ dynamic = ["version"]
 requires-python = ">=3.10"
 
 dependencies = [
-    "ckanapi>=4.8",
+    "ckanapi>=4.11",
     "defopt>=7.0.0",
     "email_validator",
     "hdx-python-country>=4.1.1",

src/hdx/data/resource.py

@@ -62,6 +62,9 @@ def actions() -> dict[str, str]:
             "search": "resource_search",
             "broken": "hdx_mark_broken_link_in_resource",
             "datastore_delete": "datastore_delete",
+            "datastore_create": "datastore_create",
+            "datastore_insert": "datastore_insert",
+            "datastore_upsert": "datastore_upsert",
             "datastore_search": "datastore_search",
         }
 
@@ -701,6 +704,53 @@ def has_datastore(self) -> bool:
                 return True
         return False
 
+    def create_datastore(
+        self,
+        schema: Sequence[dict],
+        primary_key: str | Sequence[str] | None = None,
+    ) -> None:
+        """Create a datastore for the resource with the given schema.
+
+        Args:
+            schema: Sequence of field definitions, each a dict with 'id' and 'type' keys.
+            primary_key: Primary key field name(s). Defaults to None.
+
+        Returns:
+            None
+        """
+        data: dict = {
+            "resource_id": self.data["id"],
+            "force": True,
+            "fields": schema,
+        }
+        if primary_key is not None:
+            if not isinstance(primary_key, str):
+                primary_key = ",".join(primary_key)
+            data["primary_key"] = primary_key
+        self._write_to_hdx("datastore_create", data, "resource_id")
+
+    def update_datastore(
+        self,
+        records: Sequence[dict],
+        method: str = "upsert",
+    ) -> None:
+        """Update (upsert) records into the resource datastore.
+
+        Args:
+            records: Sequence of record dicts to insert or update.
+            method: Datastore update method ('upsert', 'insert', or 'update'). Defaults to 'upsert'.
+
+        Returns:
+            None
+        """
+        data = {
+            "resource_id": self.data["id"],
+            "force": True,
+            "method": method,
+            "records": records,
+        }
+        self._write_to_hdx("datastore_upsert", data, "resource_id")
+
     def delete_datastore(self) -> None:
         """Delete a resource from the HDX datastore
 

tests/hdx/api/utilities/test_hdx_state.py

@@ -47,7 +47,7 @@ def date2(self):
     def do_state(self, tempfolder, statefile):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if "resource" in url:
                     result = json.dumps(resultdict)
                     return MockResponse(
@@ -71,7 +71,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def do_state_multi(self, tempfolder, multidatestatefile):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if "resource" in url:
                     result = json.dumps(resultdict)
                     return MockResponse(

tests/hdx/data/test_dataset_add_hapi_error.py

@@ -13,7 +13,7 @@ class TestDatasetAddHAPIError:
     def hapi_resource_update(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 if "show" in url:
                     resource_id = datadict["id"]

tests/hdx/data/test_dataset_core.py

@@ -182,7 +182,7 @@ def static_json(self, configfolder):
     def read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return dataset_mockshow(url, datadict)
 
@@ -192,7 +192,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_revise(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if isinstance(data, dict):
                     datadict = {
                         k.decode("utf8"): v.decode("utf8") for k, v in data.items()
@@ -219,7 +219,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_create(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if isinstance(data, dict):
                     datadict = {
                         k.decode("utf8"): v.decode("utf8") for k, v in data.items()
@@ -286,7 +286,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_update(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if isinstance(data, dict):
                     datadict = {
                         k.decode("utf8"): v.decode("utf8") for k, v in data.items()
@@ -363,7 +363,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_reorder(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "show" in url:
@@ -390,7 +390,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_delete(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "show" in url:
@@ -423,7 +423,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def search(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return mocksearch(url, datadict)
 
@@ -433,7 +433,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_list(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return mocklist(url, datadict)
 
@@ -443,7 +443,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def all(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return mockall(url, datadict)
 
@@ -453,7 +453,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_autocomplete(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "autocomplete" not in url or "acled" not in datadict["q"]:

tests/hdx/data/test_dataset_noncore.py

@@ -52,7 +52,7 @@ def vocabulary_read(self):
 
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return vocabulary_mockshow(url, datadict)
 
@@ -62,7 +62,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def user_read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return user_mockshow(url, datadict)
 
@@ -72,7 +72,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def organization_read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return organization_mockshow(url, datadict)
 
@@ -82,7 +82,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def showcase_read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 if "showcase_list" in url:
                     result = json.dumps([showcase_resultdict])
@@ -111,7 +111,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def vocabulary_update(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 if isinstance(data, dict):
                     datadict = {
                         k.decode("utf8"): v.decode("utf8") for k, v in data.items()

tests/hdx/data/test_organization.py

@@ -115,7 +115,7 @@ def static_json(self, configfolder):
     def read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return organization_mockshow(url, datadict)
 
@@ -125,7 +125,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_create(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 if "show" in url:
                     return organization_mockshow(url, datadict)
@@ -165,7 +165,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_update(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 if "show" in url:
                     return organization_mockshow(url, datadict)
@@ -205,7 +205,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_delete(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "show" in url:
@@ -232,7 +232,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_list(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 json.loads(data.decode("utf-8"))
                 return mocklist(url)
 
@@ -242,7 +242,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_all_fields(self, fixturesfolder):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 kwargs = json.loads(data.decode("utf-8"))
                 if "show" in url:
                     return organization_mockshow(url, kwargs)
@@ -264,7 +264,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def user_read(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return user_mockshow(url, datadict)
 
@@ -274,7 +274,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def datasets_get(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 datadict = json.loads(data.decode("utf-8"))
                 return mockgetdatasets(url, datadict)
 
@@ -284,7 +284,7 @@ def post(url, data, headers, files, allow_redirects, auth=None):
     def post_autocomplete(self):
         class MockSession:
             @staticmethod
-            def post(url, data, headers, files, allow_redirects, auth=None):
+            def post(url, data, **kwargs):
                 decodedata = data.decode("utf-8")
                 datadict = json.loads(decodedata)
                 if "autocomplete" not in url or "innago" not in datadict["q"]: