diff --git a/CHANGELOG.md b/CHANGELOG.md
index 18b67f6..5fdc2aa 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,7 +5,117 @@ All notable changes to this project are documented here.
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-## [Unreleased]
+## [1.2.0] - 2026-08-17
+
+### Added
+
+- `Explain Query` reports how MongoDB answered a query: the winning plan's stage, the
+ index it used, whether it read the collection rather than an index, how many keys and
+ documents it examined, and — the field it exists for — `index_bounds`, the values the
+ server actually searched for. A query that matches nothing and reports no error is
+ nearly always a query that asked for something other than what the suite meant, and
+ comparing the bounds with what was passed is the whole diagnosis:
+
+ ```robotframework
+ ${plan} Explain Query collection_name=readings _id.deviceId=${device_id} _id.date=${date}
+ Log ${plan.index_bounds}
+ ```
+
+ It takes the query either way the find keywords take it, as free arguments like
+ `Find Document` or as a `query` document like `Find Document With Query`, so it can be
+ dropped in beside a failing call without rewriting it. The summary is returned, and
+ logged at INFO a field to a line, with each indexed field's bounds on a line of its own:
+ Robot Framework keeps newlines and indentation in `log.html`, and a summary on one line
+ is readable by nobody. The whole explain document is logged at DEBUG, indented, and
+ returned under `raw` for anything the summary leaves out.
+ `verbosity` selects `executionStats`, which is the default and runs the winning plan,
+ `queryPlanner`, which picks a plan without running it, or `allPlansExecution`, which
+ adds the rejected plans.
+
+ The summary is derived rather than copied, because the explain document's shape differs
+ by server version and topology while the questions do not: a slot-based plan nests its
+ stages under `queryPlan`, a sharded cluster reports per shard, and the `_id` fast path
+ is `EXPRESS_IXSCAN` on MongoDB 8 and `IDHACK` before it. A collection scan is detected
+ by the presence of a `COLLSCAN` rather than by an allowlist of index stage names, so no
+ version check is involved.
+
+ The keyword is a diagnostic and asserts nothing, `collection_scan` included. MongoDB
+ rightly chooses a scan on a small collection, where reading it beats an index lookup
+ plus a fetch, so an assertion that no scan happens passes against production-sized data
+ and fails against a freshly seeded test collection with nothing wrong.
+
+- `Collection Should Have Index` fails unless an index on exactly the given fields exists,
+ reading the collection's index definitions rather than a query plan, so it cannot flake:
+
+ ```robotframework
+ Collection Should Have Index collection_name=readings keys={"_id.deviceId": 1, "_id.date": 1}
+ ```
+
+ Asks by fields rather than by name, which `Check Index Exists` does. The name is derived
+ from the fields — `_id.deviceId_1__id.date_1` — so naming it means writing out a string
+ nobody should have to spell and that changes if the index is ever recreated slightly
+ differently; the fields are what the queries depend on. The order of the fields is
+ compared, because a compound index serves its fields left to right and the same fields
+ in the other order are a different index. Failure lists every index the collection does
+ have, with its keys.
+
+ This is the guard for an index a suite quietly depends on. Dropping it breaks nothing
+ visibly: the queries still return the right documents, by reading the whole collection
+ to do it, and the suite gets slower until something times out somewhere unrelated.
+
+- Documentation for the empty result a collection keyed by a compound `_id` produces three
+ ways, none of which errors: the automatic `_id_` index cannot serve a query on part of
+ the id, since it stores the subdocument as one opaque value; matching the whole `_id`
+ compares the stored BSON and is therefore field-order sensitive; and a date compares
+ exactly, so a `datetime` at midnight never matches a document stored with milliseconds.
+
+- `Load Document` reads a seed document from a JSON file, and `Insert Document From File`
+ reads one and inserts it in a single step. A new `document_path` import argument names
+ the directory that documents given by file name are looked up in; a path given to the
+ keyword works with or without it, so the argument only removes the repetition.
+
+ The file is read as MongoDB Extended JSON, so `$oid`, `$date`, `$numberInt` and
+ `$numberDouble` become `ObjectId`, `datetime`, `int` and `float`, and the document is
+ stored with the types MongoDB compares against rather than with text that silently
+ matches nothing. Plain JSON values keep their own types. MongoDB's update operators are
+ `$`-prefixed as well and are passed through untouched, nested values included, so a file
+ can hold `{"$set": ..., "$push": ...}` for an update as readily as a document to insert.
+
+ A file can hold two kinds of hole, written differently because they are filled from
+ different places. `${name}` is a Robot Framework variable and is replaced from the ones
+ the calling suite can see, using Robot Framework's own substitution, so a value a whole
+ suite shares is written once. `{name}` is a template placeholder and is filled from the
+ loading keyword's named arguments, so a value that differs on every call is given at the
+ call:
+
+ ```robotframework
+ ${document} Load Document order.json unique_id=order-1 customerId=${oid}
+ ${document} Load Document order.json &{placeholders}
+ ```
+
+ Placeholders are written quoted — `"customerId": "{customerId}"` — which keeps the
+ template valid JSON, so editors, `jq` and formatters still read it. A string that is
+ exactly one placeholder is replaced whole, its quotes included, by the value's own
+ Extended JSON form, so an ObjectId, a datetime or a number arrives as itself with no
+ `$oid` or `$date` wrapper needed; a placeholder inside a longer string, as in
+ `"REF-{unique_id}"`, is interpolated as text. Inside a string `{{` and `}}` are literal
+ braces as in `str.format`, and JSON's own braces are never touched. Either kind of hole
+ left unfilled fails the keyword, because the literal text is a perfectly insertable
+ string that would seed a document looking almost right.
+
+ Any field the file already fills can be overridden by its dotted path, list positions
+ included, as in `lines.0.quantity=3`. This is the part a suite cannot do for itself:
+ `&{dict}` expansion merges one level deep, so overriding a nested field otherwise means
+ rebuilding every level above it. Overrides written literally are read like the file's own
+ values, so a number stays a number. A path that does not exist in the document fails with
+ what the document held at that point, because a path that misses is a typo far more often
+ than it is a field meant to be added.
+
+ Placeholders and overrides are given the same way and the file decides which an argument
+ is: a name it declares as a placeholder fills that placeholder, and anything else is a
+ path. So a value that always varies becomes a hole in the template, a value that varies
+ occasionally overrides what the file already says, and a bare name that is neither fails
+ naming both.
## [1.1.0] - 2026-08-13
diff --git a/MongoDBLibrary/__init__.py b/MongoDBLibrary/__init__.py
index 2909a6b..67ddf54 100644
--- a/MongoDBLibrary/__init__.py
+++ b/MongoDBLibrary/__init__.py
@@ -1,4 +1,5 @@
from importlib.metadata import PackageNotFoundError, version
+from typing import Optional
from robotlibcore import DynamicCore
@@ -23,6 +24,8 @@ class MongoDBLibrary(DynamicCore):
- Introduction
- Usage
- Object Ids
+ - Documents From Files
+ - Diagnosing An Empty Result
- Resetting Between Tests
- Writing A Fixture That Can Run Twice
- Hosted Clusters
@@ -112,6 +115,201 @@ class MongoDBLibrary(DynamicCore):
With the option off, nothing about your queries is altered and passing a string
``_id`` against an ObjectId-keyed collection will silently match nothing again.
+ == Documents From Files ==
+
+ A fixture document written into a suite is fine until a second test needs it. Then it
+ is copied, and the two copies drift. `Load Document` reads one from a JSON file
+ instead, and `Insert Document From File` reads it and inserts it in one step:
+
+ | Library MongoDBLibrary document_path=${CURDIR}/documents
+
+ | ${document} Load Document order.json
+ | ${doc_id} Insert Document From File collection_name=orders path=order.json
+
+ ``document_path`` only removes the repetition of naming the directory in every call. A
+ path given to the keyword is used as written whether it is set or not, so
+ ``Load Document ${CURDIR}/documents/order.json`` works without it.
+
+ === Extended JSON ===
+
+ The file is read as MongoDB Extended JSON, which is how MongoDB itself writes types
+ that JSON has no syntax for. So a document id in a file is a real ``ObjectId`` and a
+ timestamp is a real ``datetime``:
+
+ | {
+ | "_id": {"$oid": "6a7ccdea6abf6a4ebbc3514f"},
+ | "placedAt": {"$date": "2026-03-01T09:30:00Z"},
+ | "quantity": {"$numberInt": "3"},
+ | "total": {"$numberDouble": "42.50"},
+ | "status": "new",
+ | "lines": [{"sku": "A-1", "quantity": 2}]
+ | }
+
+ This matters for exactly the reason `Object Ids` describes: a string that looks like an
+ id does not match one, and a date written as text is stored as text and does not
+ compare as a date. Plain JSON values are left as the types they already are, so only
+ the fields that need a BSON type are written this way.
+
+ MongoDB's update operators start with ``$`` as well, and are not affected — they are
+ passed through as they are, nested values included. A file can therefore hold an
+ update rather than a document:
+
+ | {"$set": {"status": "shipped", "shippedAt": {"$date": "2026-03-02T00:00:00Z"}}}
+
+ | ${update} Load Document ship.json
+ | Update Documents With Operators collection_name=orders query={"status": "new"} update=${update}
+
+ === Suite variables ===
+
+ ``${...}`` in the file is replaced from the variables the calling suite can see, so a
+ value the whole suite shares is written in one place:
+
+ | {"email": "${EMAIL}", "signedUpAt": {"$date": "${SIGNED_UP_AT}"}}
+
+ A variable that resolves to nothing fails the keyword, naming the file and the
+ variable. That is the point of substituting here rather than in the suite: a document
+ that kept the literal text ``${EMAIL}`` inserts perfectly well, and the test then fails
+ somewhere later against data that looks almost right.
+
+ === Filling a template ===
+
+ A value that differs on *every* call belongs in the call, not in a suite variable. A
+ file can declare a hole for one, written ``{name}`` and filled from the keyword's named
+ arguments:
+
+ | {
+ | "unique_id": "{unique_id}",
+ | "customerId": "{customerId}",
+ | "placedAt": "{placed_at}",
+ | "quantity": "{quantity}",
+ | "reference": "REF-{unique_id}",
+ | "status": "new"
+ | }
+
+ | ${document} Load Document order.json unique_id=order-1 customerId=${oid}
+ | ... placed_at=${now} quantity=3
+
+ The template stays valid JSON, so an editor, ``jq`` and a formatter still read it. A
+ dictionary can be expanded into the arguments with ``&{placeholders}``, and an empty one
+ fills nothing.
+
+ Two different holes, then, and the syntax says which is which: ``${name}`` comes from
+ the suite, ``{name}`` from the call.
+
+ A hole is one the *file* is written with. Substitution happens first, so a variable
+ whose value contains braces — ``${GREETING}`` holding ``Hi {first_name}`` — is data:
+ the braces are inserted as they are and no argument fills them.
+
+ As for what a filled value becomes: a string that is *exactly* one placeholder is
+ replaced whole, quotes included, by the
+ value's own Extended JSON form. That is what lets a valid-JSON template carry something
+ JSON cannot write, and it means no ``$oid`` or ``$date`` wrapper is needed for a value
+ that already is one:
+
+ | "customerId": "{customerId}" with an ObjectId -> a real ObjectId
+ | "placedAt": "{placed_at}" with a datetime -> a real datetime
+ | "quantity": "{quantity}" with 3 -> a real int
+
+ A value written literally is read the way the file's own values are, so ``quantity=3``
+ is a number and ``status=shipped`` is text. A placeholder inside a longer string, as in
+ ``"REF-{unique_id}"``, is interpolated as text instead. Where a document genuinely
+ holds braces in a string, ``{{`` and ``}}`` are literal ones, as in Python's
+ ``str.format``; JSON's own braces are never touched.
+
+ A placeholder the file declares and no argument fills is an error, naming the file and
+ the holes, for the same reason an unresolved ``${...}`` is: the literal text
+ ``{unique_id}`` is a perfectly insertable string.
+
+ === Overriding fields ===
+
+ A field the file already fills can be changed without the file declaring a hole for it,
+ which is what a value that varies only *occasionally* wants — the file's own value stays
+ as the default for every test that does not mention it. Any field can be overridden by
+ its path, and a list position is written as a number:
+
+ | ${document} Load Document order.json status=shipped lines.0.quantity=3
+ | ${document} Load Document order.json customer._id=${customer_id}
+
+ This is the part a suite cannot do for itself. Robot Framework's ``&{dict}`` expansion
+ merges one level deep, so overriding a nested field means rebuilding every level above
+ it by hand.
+
+ A value written literally is read the way the file's own values are: ``0.8`` is a
+ number, ``${True}`` and ``true`` are booleans, ``{"$oid": "..."}`` is an ObjectId, and
+ a word such as ``shipped`` is the text it looks like. A value given as a variable is
+ used as it is.
+
+ Every step of a path has to exist in the document already. A path that does not fails
+ with what the document held at that point, because a path that misses is a typo far
+ more often than it is a field meant to be added — and a silent insert would leave the
+ document with both the misspelled field and the original one.
+
+ === Which one an argument is ===
+
+ Placeholders and overrides are given the same way, and the file decides which an
+ argument is: a name the file declares as a placeholder fills it, and anything else is a
+ path into the document.
+
+ | ${document} Load Document order.json unique_id=order-1 status=shipped lines.0.quantity=3
+ | # a hole the file a field it a nested field
+ | # declares already fills
+
+ A dotted name can only ever have been a path. A bare one that is neither a placeholder
+ nor a field fails naming both, because which was meant decides whether the fix belongs
+ in the file or in the call.
+
+ == Diagnosing An Empty Result ==
+
+ A query that matches nothing and reports no error is this library's most common
+ failure, and `Object Ids` covers only one cause of it. `Explain Query` covers the
+ rest: it asks the server how it answered the query and returns a summary of the plan,
+ including ``index_bounds`` — the values it actually searched for.
+
+ | ${plan} Explain Query collection_name=readings _id.deviceId=${device_id} _id.date=${date}
+ | Log ${plan.index_bounds}
+
+ It asserts nothing and is not meant to stay in a passing test. Put it beside the find
+ keyword that returned nothing, read the log, and take it out again.
+
+ === A compound id ===
+
+ A collection whose ``_id`` is a subdocument rather than a single value produces three
+ silent empty results at once, and they are worth naming because none of them errors.
+
+ | {"_id": {"deviceId": "device-1", "date": ISODate("2026-01-08T00:00:00.001Z")}}
+
+ First, the automatic ``_id_`` index cannot answer a query on part of that id. It
+ indexes the whole subdocument as one opaque value, so ``_id.deviceId`` and ``_id.date``
+ are served by a separate index if one exists and by reading every document if not. The
+ index is invisible in the suite that depends on it, which is what
+ `Collection Should Have Index` is for:
+
+ | Collection Should Have Index collection_name=readings keys={"_id.deviceId": 1, "_id.date": 1}
+
+ Second, matching the whole ``_id`` at once compares the stored BSON, so the *order* of
+ the fields is part of the value. These are two different queries and the second one
+ matches nothing:
+
+ | query={"_id": {"deviceId": "device-1", "date": ${date}}} # matches
+ | query={"_id": {"date": ${date}, "deviceId": "device-1"}} # matches nothing
+
+ Third, a date compares exactly. A ``datetime`` at midnight does not match a document
+ stored with milliseconds, which is the difference ``index_bounds`` shows:
+
+ | '_id.date': ['[new Date(1767830400000), new Date(1767830400000)]']
+
+ Comparing that with what the suite passed is the whole diagnosis.
+
+ === Reading the summary ===
+
+ ``collection_scan`` says whether the plan read the collection rather than an index.
+ Treat it as information for a person, never as something to assert on: MongoDB
+ correctly chooses a scan on a small collection, where reading it beats an index lookup
+ plus a fetch, so an assertion that no scan happens passes against production-sized
+ data and fails against a freshly seeded test collection with nothing wrong. When a
+ suite needs an index to exist, `Collection Should Have Index` says so directly and
+ cannot flake, because it reads the collection's index definitions rather than a plan.
+
== Resetting Between Tests ==
There are two ways to clear a collection and they are not interchangeable.
@@ -254,6 +452,9 @@ class MongoDBLibrary(DynamicCore):
taking the same free query parameters as `Find Document`.
- `Check Collection Exists` and `Check Index Exists` — for asserting that a migration
or an application's start-up created what it was supposed to.
+ - `Collection Should Have Index` — the same question about an index asked by its
+ fields rather than its name, which is what a suite's queries actually depend on.
+ See `Diagnosing An Empty Result`.
=== Retrying ===
@@ -292,13 +493,17 @@ class MongoDBLibrary(DynamicCore):
The keywords cover what a test suite normally needs, which is a small part of what
MongoDB can do. `Run Database Command` reaches the rest — server statistics, storage
- sizes, query plans and the administrative commands are all database commands, and
- there are far too many to give each a keyword:
+ sizes and the administrative commands are all database commands, and there are far too
+ many to give each a keyword:
| ${stats} Run Database Command command={"collStats": "orders"}
- | ${plan} Run Database Command command={"explain": {"find": "orders", "filter": {"status": "new"}}}
| ${info} Run Database Command command={"listCollections": 1}
+ A find is explained by `Explain Query` rather than here; this is the way to explain
+ something else, such as an aggregation pipeline:
+
+ | ${plan} Run Database Command command={"explain": {"aggregate": "orders", "pipeline": [], "cursor": {}}}
+
A command written as a document is read as one; anything else is sent as a bare
command name, so ``command=ping`` works too.
@@ -311,18 +516,24 @@ class MongoDBLibrary(DynamicCore):
# Read from the installed package so the version lives in pyproject.toml only.
ROBOT_LIBRARY_VERSION = __version__
- def __init__(self, coerce_object_ids: bool = True) -> None:
+ def __init__(self, coerce_object_ids: bool = True, document_path: Optional[str] = None) -> None:
"""Initializes the MongoDB Library.
Arguments:
- ``coerce_object_ids``: Whether a string ``_id`` in a query is converted to a
BSON ObjectId, on by default. See `Object Ids` for exactly what this rewrites,
why it is on, and the one case in which you want it off.
+ - ``document_path``: Directory that documents given to `Load Document` and
+ `Insert Document From File` by file name are looked up in. It saves naming the
+ directory in every call and does nothing else: a path given to the keyword is
+ used as written whether this is set or not. See `Documents From Files`.
| Library MongoDBLibrary
| Library MongoDBLibrary coerce_object_ids=${False}
+ | Library MongoDBLibrary document_path=${CURDIR}/documents
"""
self.connection_manager = ConnectionManager()
- libraries = [MongoDBKeywords(self.connection_manager, coerce_object_ids=coerce_object_ids)]
+ libraries = [MongoDBKeywords(self.connection_manager, coerce_object_ids=coerce_object_ids,
+ document_path=document_path)]
DynamicCore.__init__(self, libraries)
diff --git a/MongoDBLibrary/documents.py b/MongoDBLibrary/documents.py
new file mode 100644
index 0000000..2779839
--- /dev/null
+++ b/MongoDBLibrary/documents.py
@@ -0,0 +1,462 @@
+"""Reading seed documents from files.
+
+Kept out of ``keywords.py`` because none of it touches a connection: resolving a file
+name, substituting Robot Framework variables, filling template placeholders, reading
+Extended JSON and applying dotted overrides are all functions of a file and its arguments.
+
+A document file has two kinds of hole, and they are deliberately written differently
+because they are filled from different places. ``${name}`` is a Robot Framework variable,
+so it comes from the suite; ``{name}`` is a placeholder, so it comes from the arguments of
+the call that loads the document. Reading the file tells you which.
+"""
+
+import copy
+import json
+import re
+from pathlib import Path
+from typing import Any, NamedTuple, Optional
+
+from bson import json_util
+from robot.libraries.BuiltIn import BuiltIn, RobotNotRunningError
+
+# What makes text worth handing to Robot Framework for substitution. ``${...}`` is the
+# form documents are written with; ``%{...}`` is the environment variable form, accepted
+# because a fixture that reads a value from the environment is written the same way.
+VARIABLE_MARKERS = ("${", "%{")
+
+# A placeholder, and the same thing with the quotes that make it a whole JSON string. The
+# name is restricted to an identifier so that JSON's own braces can never match: ``{}``,
+# ``{"sku": "A-1"}`` and ``{"$set": ...}`` all fail the pattern on their first character.
+PLACEHOLDER = re.compile(r"\{([A-Za-z_]\w*)\}")
+QUOTED_PLACEHOLDER = re.compile(r'"\{([A-Za-z_]\w*)\}"')
+
+# Distinguishes "no value was given for this placeholder" from a value of None, which is a
+# legitimate thing to fill a field with.
+_MISSING = object()
+
+
+class FilledDocument(NamedTuple):
+ """The result of filling a template: the text, and what was found while filling it."""
+
+ text: str
+ used: set[str]
+ declared: list[str]
+
+
+def resolve_document_file(path: str, document_path: Optional[Path]) -> Path:
+ """
+ Return the file ``path`` names, looking in ``document_path`` first.
+
+ A bare file name is resolved against ``document_path``, which is the only thing that
+ argument does. A path given to the keyword is still tried as written, so a suite that
+ imports without ``document_path`` keeps working, and so does one that gives a path
+ reaching outside the directory.
+
+ :param path: File name or path as given to the keyword
+ :param document_path: Directory from the library import, or None
+ :return: The file that was found
+ """
+ candidates = [Path(path)]
+ if document_path is not None and not Path(path).is_absolute():
+ candidates.insert(0, document_path / path)
+ for candidate in candidates:
+ if candidate.is_file():
+ return candidate
+ searched = ", ".join(str(candidate) for candidate in candidates)
+ raise ValueError(f"Document file '{path}' was not found. Looked in: {searched}.")
+
+
+def declared_placeholders(text: str) -> set[str]:
+ """
+ Return the placeholder names ``text`` contains, for use as an allowlist.
+
+ Read from the file *before* variables are substituted, so a ``${...}`` value that
+ happens to contain braces is data rather than a template. Deliberately looser than
+ `fill_placeholders`: it ignores whether a match sits inside a string and whether the
+ braces were escaped, so it can name more than the scanner would fill. That is safe
+ because it only ever permits — the exactness stays where the filling happens.
+
+ :param text: File contents, as read
+ :return: The names the file declares
+ """
+ return {match.group(1) for match in PLACEHOLDER.finditer(text)}
+
+
+def substitute_variables(text: str, name: str) -> Any:
+ """
+ Replace the ``${...}`` placeholders in ``text`` from the calling suite's variables.
+
+ Substitution is Robot Framework's own, so the variable scope is the one the suite
+ sees at the moment the keyword runs, and a placeholder that resolves to nothing is an
+ error rather than a literal ``${user_id}`` written into the database. That distinction
+ is the reason this is not left to `Replace Variables` in the suite: a document seeded
+ with the placeholder text still inserts, and the test then fails somewhere later
+ against data that looks almost right.
+
+ :param text: File contents
+ :param name: File name, for the failure message
+ :return: The text with placeholders replaced, or the object a whole-file placeholder
+ resolved to
+ """
+ if not any(marker in text for marker in VARIABLE_MARKERS):
+ return text
+ try:
+ return BuiltIn().replace_variables(text)
+ except RobotNotRunningError as error:
+ raise ValueError(
+ f"Document '{name}' uses ${{...}} variables, which can only be resolved while a "
+ f"Robot Framework test is running."
+ ) from error
+ except Exception as error:
+ raise ValueError(f"Document '{name}' has a variable that could not be resolved: {error}") from error
+
+
+def fill_placeholders(text: str, values: dict[str, Any], name: str,
+ allowed: Optional[set[str]] = None) -> FilledDocument:
+ """
+ Fill the ``{name}`` placeholders of a template document from ``values``.
+
+ A placeholder is written quoted, as ``"quantity": "{quantity}"``, which keeps the
+ template valid JSON — an editor, ``jq`` and a formatter all still read it — and it is
+ filled in one of two ways depending on where it sits:
+
+ - A string that is *exactly* one placeholder is replaced whole, its quotes included, by
+ the value's own Extended JSON form. This is what lets a valid-JSON template carry a
+ value JSON has no syntax for: ``"quantity": "{quantity}"`` given ``3`` becomes a real
+ int, and an ObjectId or a datetime becomes ``{"$oid": ...}`` or ``{"$date": ...}`` and
+ parses back to what it was.
+ - A placeholder inside a longer string, as in ``"REF-{n}"``, is interpolated as text.
+
+ An unquoted ``{name}`` in a value position is filled like the first case. It makes the
+ template invalid JSON before it is filled, which is why the quoted form is the one
+ documented, but nothing here needs the file to parse yet, so it works.
+
+ Inside a string ``{{`` and ``}}`` are literal braces, as in ``str.format``, which is how
+ a document that genuinely holds ``{word}`` in a string says so. Outside a string they
+ are JSON's own braces and are left alone — unescaping there would rewrite the ``}}``
+ that closes every nested object in the file.
+
+ A placeholder is something the file says, so ``allowed`` names the placeholders that
+ were in the file as it was read. A ``${...}`` variable is substituted before this runs,
+ and a variable's value is data: braces that arrive that way are left exactly as they
+ came, never filled and never reported as an unfilled hole the file never had.
+
+ :param text: File contents, ``${...}`` variables already substituted
+ :param values: Candidate values, which are the loading keyword's named arguments
+ :param name: File name, for the failure message
+ :param allowed: Placeholder names the file itself declares, from `declared_placeholders`
+ on the unsubstituted text, or None to fill whatever the text now contains
+ :return: The filled text, the keys of ``values`` it used, and the placeholders the
+ file declares
+ """
+ filled: list[str] = []
+ used: set[str] = set()
+ declared: list[str] = []
+ missing: list[str] = []
+
+ def from_file(key: str) -> bool:
+ """Whether ``key`` is a placeholder the file declared, rather than substituted text."""
+ return allowed is None or key in allowed
+
+ def take(key: str) -> Any:
+ """Record a placeholder as declared, and return its value if there is one."""
+ if key not in declared:
+ declared.append(key)
+ if key in values:
+ used.add(key)
+ return values[key]
+ if key not in missing:
+ missing.append(key)
+ return _MISSING
+
+ index = 0
+ in_string = False
+ while index < len(text):
+ character = text[index]
+ if not in_string:
+ quoted = QUOTED_PLACEHOLDER.match(text, index)
+ if quoted is not None and from_file(quoted.group(1)): # a whole string, so the value keeps its own type
+ value = take(quoted.group(1))
+ filled.append(quoted.group(0) if value is _MISSING else _as_json(value))
+ index = quoted.end()
+ continue
+ if character == '"':
+ in_string = True
+ filled.append(character)
+ index += 1
+ continue
+ bare = PLACEHOLDER.match(text, index)
+ if bare is not None and from_file(bare.group(1)): # unquoted, so it is a value position too
+ value = take(bare.group(1))
+ filled.append(bare.group(0) if value is _MISSING else _as_json(value))
+ index = bare.end()
+ continue
+ filled.append(character)
+ index += 1
+ continue
+ if character == "\\": # an escape, whose second character is never a brace or quote
+ filled.append(text[index:index + 2])
+ index += 2
+ continue
+ if character == '"':
+ in_string = False
+ filled.append(character)
+ index += 1
+ continue
+ if text.startswith("{{", index) or text.startswith("}}", index):
+ filled.append(character)
+ index += 2
+ continue
+ inside = PLACEHOLDER.match(text, index)
+ if inside is not None and from_file(inside.group(1)): # part of a longer string, so the value is text here
+ value = take(inside.group(1))
+ filled.append(inside.group(0) if value is _MISSING else _as_json_string_body(value))
+ index = inside.end()
+ continue
+ filled.append(character)
+ index += 1
+
+ if missing:
+ _reject_unfilled(missing, values, name)
+ return FilledDocument("".join(filled), used, declared)
+
+
+def _as_json(value: Any) -> str:
+ """Render a value as the Extended JSON that parses back to it."""
+ return json_util.dumps(coerce_override(value))
+
+
+def _as_json_string_body(value: Any) -> str:
+ """Render a value as text to sit inside a JSON string, escaped for one."""
+ return json_util.dumps(str(value))[1:-1]
+
+
+def _reject_unfilled(missing: list[str], values: dict[str, Any], name: str) -> None:
+ """
+ Fail on a placeholder the file declares that no argument filled.
+
+ Loud rather than left as it is, for the same reason an unresolved ``${...}`` fails: the
+ literal text ``{unique_id}`` is a perfectly insertable string, so leaving it would put a
+ document in the database that looks almost right and fail a test somewhere later.
+ """
+ holes = ", ".join(f"{{{key}}}" for key in missing)
+ given = ", ".join(sorted(values)) or "nothing"
+ was = "were" if len(missing) > 1 else "was"
+ raise ValueError(
+ f"Document '{name}' declares {holes}, which {was} not filled. Pass a value as a "
+ f"named argument for each. Given: {given}."
+ )
+
+
+def parse_document(text: str, name: str) -> dict:
+ """
+ Read MongoDB Extended JSON, so the document holds the types MongoDB stores.
+
+ ``bson.json_util`` turns ``$oid``, ``$date``, ``$numberInt`` and ``$numberDouble``
+ into ``ObjectId``, ``datetime``, ``int`` and ``float``, and leaves plain JSON values
+ as the types they already are. MongoDB's update operators are ``$``-prefixed too and
+ are not extended types, so ``{"$set": ..., "$push": ...}`` passes through untouched —
+ including a ``$date`` nested inside one — which is what makes this usable for update
+ documents as well as for documents to insert.
+
+ :param text: JSON text, variables and placeholders already replaced
+ :param name: File name, for the failure message
+ :return: The document
+ """
+ try:
+ document = json_util.loads(text)
+ except json.JSONDecodeError as error:
+ raise ValueError(
+ f"Document '{name}' is not valid JSON: {error.msg}, at line {error.lineno} column {error.colno}."
+ ) from error
+ except Exception as error:
+ # A converter rejected a value: a literal ``$date`` holding something that is not
+ # a date, an ``$oid`` that is not an object id. Its own message names the value,
+ # which is what identifies the field, so it is reported rather than replaced.
+ raise ValueError(f"Document '{name}' has a value Extended JSON could not read: {error}") from error
+ if not isinstance(document, dict):
+ raise ValueError(
+ f"Document '{name}' has to hold a JSON object, but it holds a {type(document).__name__}."
+ )
+ return document
+
+
+def as_document(value: Any, name: str) -> dict:
+ """
+ Return the document ``value`` describes, parsing it when it is still text.
+
+ Text is the normal case. An object turns up when the whole file is a single variable,
+ such as a file holding nothing but ``${ORDER}``: Robot Framework resolves that to the
+ variable itself rather than to its printed form. It is copied, because overrides are
+ applied in place and the suite's variable is not this keyword's to change.
+
+ :param value: Result of substitution
+ :param name: File name, for the failure message
+ :return: The document
+ """
+ if isinstance(value, dict):
+ return copy.deepcopy(value)
+ return parse_document(str(value), name)
+
+
+def build_document(substituted: Any, arguments: dict[str, Any], name: str,
+ allowed: Optional[set[str]] = None) -> dict:
+ """
+ Turn a substituted file into the finished document, using ``arguments`` for both jobs.
+
+ An argument is one of two things and the file decides which: a key the file declares as
+ a placeholder fills that placeholder, and any other key is a dotted override path into
+ the parsed document. So a value that changes on every call is a hole in the template,
+ while a value that changes occasionally overrides what the file already says, and
+ neither needs the other to have been thought of first.
+
+ :param substituted: Result of `substitute_variables`
+ :param arguments: The loading keyword's named arguments
+ :param name: File name, for the failure messages
+ :param allowed: Placeholder names the file declares, from `declared_placeholders` on
+ the text before substitution, or None to fill whatever the substituted text holds
+ :return: The document
+ """
+ declared: list[str] = []
+ remaining = arguments
+ if isinstance(substituted, str):
+ filled = fill_placeholders(substituted, arguments, name, allowed)
+ substituted, declared = filled.text, filled.declared
+ remaining = {key: value for key, value in arguments.items() if key not in filled.used}
+ document = as_document(substituted, name)
+ _reject_unknown_arguments(remaining, document, declared, name)
+ return apply_overrides(document, remaining, name)
+
+
+def _reject_unknown_arguments(arguments: dict[str, Any], document: dict, declared: list[str],
+ name: str) -> None:
+ """
+ Fail on a bare argument that is neither a placeholder in the file nor a field to override.
+
+ A dotted argument can only ever have been a path, and reports itself as one while it is
+ walked. A bare name could have been meant as either, and which was meant decides whether
+ the fix belongs in the file or in the call, so the failure names both.
+ """
+ for key in arguments:
+ if "." in key or key in document:
+ continue
+ placeholders = ", ".join(f"{{{item}}}" for item in declared) or "none"
+ fields = ", ".join(document) or "nothing"
+ raise ValueError(
+ f"Document '{name}': '{key}' is neither a placeholder the file declares nor a "
+ f"field of the document. Placeholders: {placeholders}. Fields: {fields}."
+ )
+
+
+def coerce_override(value: Any) -> Any:
+ """
+ Read an override value that was written in a suite as text.
+
+ Robot Framework passes ``evaporationFactor=0.8`` as the string ``"0.8"``, and a
+ string where the document held a number changes what MongoDB stores and how it
+ compares. Each value is read as Extended JSON, which gives numbers, booleans, null
+ and ``{"$oid": ...}`` what they mean, and leaves anything that is not JSON — an
+ ordinary word such as ``abc`` — as the text it already is. A value given as
+ ``${variable}`` arrives as an object and is used unchanged.
+
+ :param value: Override value as given to the keyword
+ :return: The value to write into the document
+ """
+ if not isinstance(value, str):
+ return value
+ try:
+ return json_util.loads(value)
+ except Exception:
+ return value
+
+
+def apply_overrides(document: dict, overrides: dict[str, Any], name: str) -> dict:
+ """
+ Write each override into ``document`` at its dotted path.
+
+ :param document: Document to modify in place
+ :param overrides: Dotted path to value, as given to the keyword
+ :param name: File name, for the failure messages
+ :return: The same document
+ """
+ for path, value in overrides.items():
+ _write_path(document, path, coerce_override(value), name)
+ return document
+
+
+def _location(walked: list[str]) -> str:
+ """Name the place a path had reached, for a failure message."""
+ return f"'{'.'.join(walked)}'" if walked else "the document"
+
+
+def _prefix(path: str, name: str) -> str:
+ """Open a failure message with the override and the file it was applied to."""
+ return f"Override '{path}' for document '{name}': "
+
+
+def _write_path(document: dict, path: str, value: Any, name: str) -> None:
+ """
+ Follow a dotted ``path`` into ``document`` and write ``value`` at the end of it.
+
+ List positions are written as numbers, as in ``components.0.evaporationFactor``, so
+ one syntax covers both objects and lists. Every step has to exist, the last one
+ included: a step that does not is reported with what was available at that point,
+ because a path that misses usually means a typo rather than a field meant to be
+ added.
+ """
+ steps = path.split(".")
+ current: Any = document
+ walked: list[str] = []
+ for step in steps[:-1]:
+ current = _read_step(current, step, walked, path, name)
+ walked.append(step)
+ _write_step(current, steps[-1], value, walked, path, name)
+
+
+def _read_step(current: Any, step: str, walked: list[str], path: str, name: str) -> Any:
+ """Take one step into a document, failing with what was there instead."""
+ prefix = _prefix(path, name)
+ if isinstance(current, dict):
+ if step not in current:
+ available = ", ".join(current) or "nothing"
+ raise ValueError(f"{prefix}'{step}' is not in {_location(walked)}. Available: {available}.")
+ return current[step]
+ if isinstance(current, list):
+ return current[_index(current, step, walked, prefix)]
+ if current is None:
+ raise ValueError(f"{prefix}{_location(walked)} is null, so '{step}' cannot be read from it.")
+ raise ValueError(
+ f"{prefix}{_location(walked)} is a {type(current).__name__}, so '{step}' cannot be read from it."
+ )
+
+
+def _write_step(current: Any, step: str, value: Any, walked: list[str], path: str, name: str) -> None:
+ """Write the last step of a path, which has to be a field the document already has."""
+ prefix = _prefix(path, name)
+ if isinstance(current, dict):
+ if step not in current:
+ available = ", ".join(current) or "nothing"
+ raise ValueError(f"{prefix}'{step}' is not in {_location(walked)}. Available: {available}.")
+ current[step] = value
+ return
+ if isinstance(current, list):
+ current[_index(current, step, walked, prefix)] = value
+ return
+ if current is None:
+ raise ValueError(f"{prefix}{_location(walked)} is null, so '{step}' cannot be set on it.")
+ raise ValueError(
+ f"{prefix}{_location(walked)} is a {type(current).__name__}, so '{step}' cannot be set on it."
+ )
+
+
+def _index(current: list, step: str, walked: list[str], prefix: str) -> int:
+ """Read a path step as a list position, checking it against the list's length."""
+ try:
+ index = int(step)
+ except ValueError:
+ raise ValueError(f"{prefix}{_location(walked)} is a list, so '{step}' has to be a number.") from None
+ if not -len(current) <= index < len(current):
+ raise ValueError(
+ f"{prefix}{_location(walked)} holds {len(current)} items, so index {step} is out of range."
+ )
+ return index
diff --git a/MongoDBLibrary/keywords.py b/MongoDBLibrary/keywords.py
index 7fbdda9..3d4af9e 100644
--- a/MongoDBLibrary/keywords.py
+++ b/MongoDBLibrary/keywords.py
@@ -1,5 +1,8 @@
+import json
import time
from ast import literal_eval
+from collections.abc import Iterator
+from pathlib import Path
from typing import Any, Callable, cast, Optional, TYPE_CHECKING, Union
from assertionengine import AssertionOperator, verify_assertion
@@ -13,6 +16,12 @@
from robot.utils import DotDict, timestr_to_secs
from MongoDBLibrary.connection_pool import ConnectionManager
+from MongoDBLibrary.documents import (
+ build_document,
+ declared_placeholders,
+ resolve_document_file,
+ substitute_variables,
+)
try: # Robot Framework 7.4 and later
from robot.api.types import Secret
@@ -32,6 +41,12 @@
Credential = Union[str, Secret] if Secret is not None else str
OptionalCredential = Optional[Credential]
+# Parts of an explain document that describe plans the server did not run: the candidates
+# it rejected, and, at ``allPlansExecution`` verbosity, the trial runs it used to choose
+# between them. Both are written in the same shape as the plan that did run, so a summary
+# that walks the whole document reports work that never happened.
+NOT_EXECUTED = ("rejectedPlans", "allPlansExecution")
+
class MongoDBKeywords:
"""
@@ -40,7 +55,8 @@ class MongoDBKeywords:
This class contains Robot Framework keywords for MongoDB operations.
"""
- def __init__(self, connection_manager: ConnectionManager, coerce_object_ids: bool = True):
+ def __init__(self, connection_manager: ConnectionManager, coerce_object_ids: bool = True,
+ document_path: Optional[str] = None):
"""
Initializes the MongoDBKeywords library.
@@ -48,9 +64,11 @@ def __init__(self, connection_manager: ConnectionManager, coerce_object_ids: boo
- ``connection_manager``: Manages connections to MongoDB.
- ``coerce_object_ids``: Whether a string ``_id`` in a query is converted to an
ObjectId.
+ - ``document_path``: Directory that documents given by file name are looked up in.
"""
self.connection_manager = connection_manager
self.coerce_object_ids = coerce_object_ids
+ self.document_path = Path(document_path) if document_path else None
# ----------------------------------------------------------------- #
# Internals
@@ -198,6 +216,137 @@ def _find(self, collection_name: str, query: Any, alias: Optional[str], projecti
cursor = cursor.sort(sort_list)
return self._as_dot_dict(list(cursor))
+ @staticmethod
+ def _explain_stages(node: Any) -> Iterator[dict]:
+ """Yield every stage in an explain document, wherever the server put it.
+
+ The plan is a tree whose links are named differently depending on what produced
+ it: ``inputStage`` and ``inputStages`` for a classic plan, ``queryPlan`` for the
+ slot-based engine, ``executionStages`` for the executed side, and ``shards`` for a
+ sharded cluster. Walking every nested dictionary and list instead of following
+ those names by hand means a shape this library has not seen still reports its
+ stages, which is the whole reason the summary can stay version-agnostic.
+
+ The walk stays agnostic about the names of the *links*, but deliberately not about
+ ``NOT_EXECUTED``: those two hold plans the server considered and did not use, and a
+ stage from one of them describes work that never happened. Yielding them would let
+ a rejected COLLSCAN report a collection scan for a query answered by an index, and
+ a rejected index plan lend its ``indexName`` to a query that scanned the collection.
+ """
+ if isinstance(node, dict):
+ if "stage" in node:
+ yield node
+ for key, value in node.items():
+ if key not in NOT_EXECUTED:
+ yield from MongoDBKeywords._explain_stages(value)
+ elif isinstance(node, list):
+ for item in node:
+ yield from MongoDBKeywords._explain_stages(item)
+
+ @staticmethod
+ def _first_stage_value(stages: list[dict], field: str) -> Any:
+ """Return ``field`` from the first stage that has it, or None."""
+ return next((stage[field] for stage in stages if field in stage), None)
+
+ @staticmethod
+ def _format_explain(collection_name: str, query: Any, summary: dict[str, Any]) -> str:
+ """Lay the summary out over several lines, aligned, for the log.
+
+ Robot Framework renders a logged message with ``white-space: pre-wrap``, so
+ newlines and indentation survive into ``log.html``. A summary on one line is
+ readable by nobody, and the field this keyword exists for — ``index_bounds`` —
+ is the one that suffers most, since it is a dictionary inside it. Here each
+ indexed field gets a line of its own, which is what makes comparing the bounds
+ with what the suite passed a matter of reading rather than of parsing.
+ """
+ lines = [f"Explain of {query!r} on '{collection_name}':"]
+ scalars = [(name, value) for name, value in summary.items() if name not in ("index_bounds", "shards")]
+ width = max(len(name) for name, _ in scalars)
+ lines += [f" {name:<{width}} {value}" for name, value in scalars]
+ bounds = summary.get("index_bounds")
+ if bounds:
+ lines.append(" index_bounds")
+ field_width = max(len(field) for field in bounds)
+ for field, values in bounds.items():
+ shown = values if isinstance(values, list) else [values]
+ lines += [f" {field:<{field_width}} {value}" for value in shown]
+ for shard in summary.get("shards") or []:
+ lines.append(f" shard {shard.get('shard')}")
+ lines += [
+ f" {name:<{width}} {value}" for name, value in shard.items() if name not in ("shard", "index_bounds")
+ ]
+ return "\n".join(lines)
+
+ @classmethod
+ def _summarise_explain(cls, raw: dict) -> dict[str, Any]:
+ """Flatten an explain document into the handful of fields a test author reads.
+
+ Everything here is derived rather than copied, because the shapes differ by
+ server version and topology while the questions do not: which plan won, what it
+ searched for, and how much it had to look at to answer.
+ """
+ winning = raw.get("queryPlanner", {}).get("winningPlan", {})
+ # The slot-based engine nests the plan the classic one puts at the top.
+ root = winning.get("queryPlan", winning)
+ stages = list(cls._explain_stages(raw))
+ execution = raw.get("executionStats", {})
+ summary: dict[str, Any] = {
+ "stage": root.get("stage"),
+ "index_name": cls._first_stage_value(stages, "indexName"),
+ "index_bounds": cls._first_stage_value(stages, "indexBounds"),
+ # Presence of a COLLSCAN, rather than a list of index stage names: the fast
+ # path for `_id` is reported as EXPRESS_IXSCAN on MongoDB 8 and IDHACK before
+ # it, and an allowlist would have to grow with every server release.
+ "collection_scan": any(stage.get("stage") == "COLLSCAN" for stage in stages),
+ "keys_examined": execution.get("totalKeysExamined"),
+ "docs_examined": execution.get("totalDocsExamined"),
+ "returned": execution.get("nReturned"),
+ "duration_ms": execution.get("executionTimeMillis"),
+ }
+ shards = cls._shard_summaries(raw)
+ if shards is not None:
+ summary["shards"] = shards
+ return summary
+
+ @classmethod
+ def _shard_summaries(cls, raw: dict) -> Optional[list[dict[str, Any]]]:
+ """Summarise each shard of a sharded plan, or None when the plan is not sharded.
+
+ A shard that reads nothing is as interesting as one that reads everything, so
+ each is reported separately rather than only as part of the total.
+ """
+ planner_shards = raw.get("queryPlanner", {}).get("winningPlan", {}).get("shards")
+ execution_shards = raw.get("executionStats", {}).get("executionStages", {}).get("shards")
+ if planner_shards is None and execution_shards is None:
+ return None
+ by_name: dict[str, dict[str, Any]] = {}
+ for shard in planner_shards or []:
+ name = shard.get("shardName")
+ plan = shard.get("winningPlan", {})
+ plan = plan.get("queryPlan", plan)
+ stages = list(cls._explain_stages(shard))
+ by_name[name] = {
+ "shard": name,
+ "stage": plan.get("stage"),
+ "index_name": cls._first_stage_value(stages, "indexName"),
+ "index_bounds": cls._first_stage_value(stages, "indexBounds"),
+ "collection_scan": any(stage.get("stage") == "COLLSCAN" for stage in stages),
+ }
+ for shard in execution_shards or []:
+ name = shard.get("shardName")
+ summary = by_name.setdefault(name, {"shard": name})
+ # The shard's own totals where it reports them, and its root stage's counts
+ # otherwise. The totals are what the unsharded summary counts, so a shard
+ # whose plan has several stages is reported the same way the whole query is.
+ stage = shard.get("executionStages", {})
+ for key, total, per_stage in (
+ ("keys_examined", "totalKeysExamined", "keysExamined"),
+ ("docs_examined", "totalDocsExamined", "docsExamined"),
+ ("returned", "nReturned", "nReturned"),
+ ):
+ summary[key] = shard.get(total, stage.get(per_stage))
+ return list(by_name.values())
+
def _retry_until_no_assertion_error(self, check: Callable[[], None], retry_timeout: str, retry_pause: str) -> None:
"""
Run ``check`` until it stops raising AssertionError or ``retry_timeout`` elapses.
@@ -490,6 +639,70 @@ def convert_to_object_id(self, value: str) -> ObjectId:
"""
return ObjectId(value)
+ # ----------------------------------------------------------------- #
+ # Documents from files
+ # ----------------------------------------------------------------- #
+
+ @keyword
+ def load_document(self, path: str, **arguments: Any) -> dict:
+ """
+ Read a document from a JSON file, ready to insert or to update with.
+
+ The file is MongoDB Extended JSON, so it can hold the types MongoDB stores rather
+ than only what plain JSON can express. See `Documents From Files` for the whole
+ picture; the short version is that the file is read in four steps:
+
+ 1. ``${...}`` variables are replaced from the ones the calling suite can see, so a
+ value shared by a whole suite is written once. One that resolves to nothing
+ fails the keyword.
+ 2. ``{...}`` placeholders are filled from this keyword's named arguments, so a
+ value that differs on every call is given at the call. One that is left
+ unfilled fails the keyword. A placeholder is one the file itself is written
+ with: braces that arrive in step 1, as part of a variable's value, are data and
+ are inserted as they are.
+ 3. ``$oid``, ``$date``, ``$numberInt`` and ``$numberDouble`` become ``ObjectId``,
+ ``datetime``, ``int`` and ``float``. Plain JSON values keep their own types,
+ and MongoDB's ``$``-prefixed update operators are left alone, so an update
+ document works here as well as a document to insert.
+ 4. Any argument that is not a placeholder is written in as a dotted override path.
+
+ Arguments:
+ - ``path``: File name, resolved against the ``document_path`` given at import, or
+ a path, used as written.
+ - ``arguments``: A value for each ``{placeholder}`` the file declares, and a dotted
+ path to a value for each field to override. Which one an argument is depends on
+ the file: a name it declares as a placeholder fills that placeholder, and
+ anything else is a path. ``ingredients.0`` is the first item of a list, so one
+ syntax covers objects and lists both. Every step of an override path has to exist
+ in the document: a path that does not is a typo far more often than it is a field
+ meant to be added, and it fails with what the document did hold at that point.
+
+ Returns:
+ - The document, as a dictionary whose fields Robot Framework can reach with
+ ``${document.field}``.
+
+ Example:
+ | ${document} Load Document order.json
+ | ${document} Load Document order.json unique_id=order-1 customerId=${customer_id}
+ | ${document} Load Document order.json &{placeholders}
+ | ${document} Load Document order.json status=shipped lines.0.quantity=3
+ | ${doc_id} Insert Document collection_name=orders document=${document}
+
+ A dictionary given as ``&{placeholders}`` is expanded into named arguments by Robot
+ Framework, so an empty one fills nothing and the file's own values stand.
+
+ See `Insert Document From File` for the last two lines written as one keyword.
+
+ """
+ document_file = resolve_document_file(path, self.document_path)
+ text = document_file.read_text(encoding="utf-8")
+ substituted = substitute_variables(text, document_file.name)
+ # From the file as it was read, so a variable whose value contains braces is data.
+ allowed = declared_placeholders(text)
+ document = build_document(substituted, arguments, document_file.name, allowed)
+ logger.debug(f"Loaded document from '{document_file}': {document!r}")
+ return cast(dict, self._as_dot_dict(document))
+
# ----------------------------------------------------------------- #
# Inserting
# ----------------------------------------------------------------- #
@@ -542,6 +755,39 @@ def insert_documents(self, collection_name: str, documents: list, alias: Optiona
collection = self._get_collection(collection_name, alias)
return collection.insert_many(documents, ordered=ordered).inserted_ids
+ @keyword
+ def insert_document_from_file(self, collection_name: str, path: str, alias: Optional[str] = None,
+ **arguments: Any) -> Any:
+ """
+ Insert a document read from a JSON file, which is `Load Document` and
+ `Insert Document` in one step.
+
+ Seeding from a file and inserting it is the shape a fixture almost always wants,
+ and the document itself is rarely worth a variable. Take it in two keywords
+ instead when the same document is inserted more than once, or when the test needs
+ the document as well as what was stored.
+
+ Arguments:
+ - ``collection_name``: Name of the collection where the document will be inserted.
+ - ``path``: File name or path, resolved exactly as `Load Document` resolves it.
+ - ``alias``: Alias of the connection (optional, defaults to the active alias).
+ - ``arguments``: Placeholder values and dotted override paths, as `Load Document`
+ takes them. A placeholder or field genuinely named ``collection_name``, ``path``
+ or ``alias`` cannot be given here, since those name this keyword's own arguments;
+ use `Load Document` and `Insert Document` for that document.
+
+ Returns:
+ - The ID of the inserted document, as `Insert Document` returns it.
+
+ Example:
+ | ${doc_id} Insert Document From File collection_name=orders path=order.json
+ | ${doc_id} Insert Document From File collection_name=orders path=order.json unique_id=order-1
+ | ${doc_id} Insert Document From File collection_name=orders path=order.json status=shipped
+
+ """
+ document = self.load_document(path, **arguments)
+ return self.insert_document(collection_name, document, alias)
+
# ----------------------------------------------------------------- #
# Reading
# ----------------------------------------------------------------- #
@@ -795,6 +1041,107 @@ def execute_query(self, collection_name: str, pipeline: list, alias: Optional[st
options: dict[str, Any] = {"allowDiskUse": True} if allow_disk_use else {}
return self._as_dot_dict(list(collection.aggregate(pipeline, **options)))
+ @keyword
+ def explain_query(self, collection_name: str, query: Optional[dict] = None, alias: Optional[str] = None,
+ projection: Optional[dict] = None, sort: Optional[dict] = None, limit: int = 0, skip: int = 0,
+ verbosity: str = "executionStats", **params: Any) -> dict:
+ """
+ Report how MongoDB answers a query: which plan it chose and what it searched for.
+
+ *This keyword is a diagnostic and asserts nothing.* It is for the moment a find
+ keyword returns nothing and gives no reason, and it is not meant to stay in a
+ passing test. Nothing here should be asserted on — see ``collection_scan`` below
+ for why the obvious assertion is the wrong one.
+
+ The field to read first is ``index_bounds``: the values the server actually
+ searched for, as it understood them.
+
+ | '_id.date': ['[new Date(1702996077710), new Date(1702996077710)]']
+
+ A query that matches nothing and errors on nothing is nearly always a query that
+ asked for something other than what the caller meant, and this is where that
+ becomes visible — a ``datetime`` at midnight against documents stored with
+ millisecond precision, or a string where the collection holds an ObjectId. Compare
+ the bounds with what the suite passed.
+
+ Takes the query either way the find keywords take it: as free arguments like
+ `Find Document`, or as a ``query`` document like `Find Document With Query`.
+ Giving both fails, so an explain can be dropped in beside a failing call unchanged.
+
+ Arguments:
+ - ``collection_name``: Name of the collection to explain the query against.
+ - ``query``: MongoDB query document, as `Find Document With Query` takes it
+ (optional). Mutually exclusive with ``params``.
+ - ``params``: Query parameters, as `Find Document` takes them. Mutually exclusive
+ with ``query``.
+ - ``alias``: Alias of the connection (optional, defaults to the active alias).
+ - ``projection``, ``sort``, ``limit``, ``skip``: The rest of the find, so that
+ what is explained is the query the suite actually runs (optional).
+ - ``verbosity``: How much the server reports (optional). ``executionStats``, the
+ default, runs the winning plan and counts what it read. ``queryPlanner`` picks a
+ plan without running it, leaving the four counters below empty.
+ ``allPlansExecution`` adds the plans that were rejected.
+
+ Returns:
+ - A dictionary summarising the plan, with the server's own explain document under
+ ``raw``. The summary is also logged at INFO, a field to a line, and the whole
+ explain document at DEBUG, indented:
+
+ | ``stage`` | The winning plan's stage, e.g. ``IXSCAN`` or ``COLLSCAN``. |
+ | ``index_name`` | Name of the index used, or None if none was. |
+ | ``index_bounds`` | The values searched for, per indexed field. |
+ | ``collection_scan`` | Whether the plan reads the collection rather than an index. |
+ | ``keys_examined`` | ``totalKeysExamined``: index entries read. |
+ | ``docs_examined`` | ``totalDocsExamined``: documents read. |
+ | ``returned`` | ``nReturned``: documents matched. |
+ | ``duration_ms`` | ``executionTimeMillis``. |
+ | ``shards`` | The same per shard, on a sharded cluster only. |
+
+ ``collection_scan`` is reported as information for a person to read, and is not
+ something to assert on. MongoDB legitimately chooses a collection scan on a small
+ collection, where reading it beats an index lookup plus a fetch, so an assertion
+ that no scan happens passes against production-sized data and fails against a
+ freshly seeded test collection with nothing wrong. To require that an index
+ exists, assert that directly with `Collection Should Have Index`.
+
+ The query is rewritten as every find keyword rewrites it, so a string ``_id`` is
+ explained as the ObjectId that would really be sent. See `Object Ids`.
+
+ Aggregation pipelines are not covered; explain one with `Run Database Command`.
+
+ Example:
+ | ${plan} Explain Query collection_name=readings _id.deviceId=${device_id} _id.date=${date}
+ | Log ${plan.index_bounds}
+ | ${plan} Explain Query collection_name=orders query={"status": "new"} sort={"placedAt": -1}
+
+ """
+ if query is not None and params:
+ raise ValueError(
+ f"Give the query either as 'query' or as free arguments, not both. "
+ f"Got query={query!r} and {sorted(params)}."
+ )
+ database = self._get_database(alias)
+ find: dict[str, Any] = {
+ "find": collection_name,
+ "filter": self._normalise_query(query if query is not None else params),
+ }
+ if projection:
+ find["projection"] = projection
+ if sort:
+ find["sort"] = sort
+ if limit:
+ find["limit"] = limit
+ if skip:
+ find["skip"] = skip
+ raw = dict(database.command({"explain": find, "verbosity": verbosity}))
+ summary = self._summarise_explain(raw)
+ logger.info(self._format_explain(collection_name, find["filter"], summary))
+ # The whole explain at DEBUG, indented. It is far too long to read every time and
+ # exactly what is wanted on the occasion the summary leaves out the answer.
+ logger.debug(json.dumps(raw, indent=2, default=str))
+ summary["raw"] = raw
+ return self._as_dot_dict(summary)
+
# ----------------------------------------------------------------- #
# Updating
# ----------------------------------------------------------------- #
@@ -1634,6 +1981,71 @@ def check() -> None:
self._retry_until_no_assertion_error(check, retry_timeout, retry_pause)
+ @keyword
+ def collection_should_have_index(
+ self,
+ collection_name: str,
+ keys: dict,
+ alias: Optional[str] = None,
+ assertion_message: Optional[str] = None,
+ retry_timeout: str = "0 seconds",
+ retry_pause: str = "0.5 seconds"
+ ) -> None:
+ """
+ Fail unless an index on exactly these fields exists on the collection.
+
+ The guard for an index a suite's queries quietly depend on. Dropping it does not
+ break anything visibly: the queries still return the right documents, by reading
+ the whole collection to do it, and the suite gets slower until something times
+ out somewhere unrelated. This turns that into one failing assertion naming the
+ index.
+
+ Asks by fields rather than by name, which `Check Index Exists` does. A name is
+ derived from the fields — an index on ``{"_id.deviceId": 1, "_id.date": 1}`` is
+ called ``_id.deviceId_1__id.date_1`` — so naming it means writing out a string
+ nobody should have to spell, that changes if the index is ever recreated slightly
+ differently. The fields are what the queries actually depend on.
+
+ The order of the fields matters and is compared: a compound index serves a query
+ on its first field, or its first two, and so on, so an index on
+ ``{"deviceId": 1, "date": 1}`` is a different index from one on
+ ``{"date": 1, "deviceId": 1}``.
+
+ Arguments:
+ - ``collection_name``: Name of the collection.
+ - ``keys``: Fields the index is on, in order, exactly as `Create Index` takes
+ them, e.g. ``{"_id.deviceId": 1, "_id.date": 1}``.
+ - ``alias``: Alias of the connection (optional, defaults to the active alias).
+ - ``assertion_message``: Custom message for assertion failure (optional).
+ - ``retry_timeout``: How long to keep checking before failing (optional). Give it
+ a value when a migration or an application's start-up creates the index.
+ - ``retry_pause``: Pause duration between retries (optional).
+
+ Example:
+ | Collection Should Have Index collection_name=readings keys={"_id.deviceId": 1, "_id.date": 1}
+ | Collection Should Have Index collection_name=users keys={"email": 1} retry_timeout=10 seconds
+
+ """
+ collection = self._get_collection(collection_name, alias)
+ expected = [(field, direction) for field, direction in keys.items()]
+
+ def check() -> None:
+ existing = dict(collection.index_information())
+ if any(list(definition.get("key", [])) == expected for definition in existing.values()):
+ return
+ # Every collection that exists has at least ``_id_``, so nothing at all means
+ # the collection does not, which is a likelier explanation of the failure than
+ # a missing index and is worth saying rather than leaving the sentence empty.
+ present = ", ".join(
+ f"{name} {dict(definition.get('key', []))}" for name, definition in sorted(existing.items())
+ ) or "no indexes at all, so the collection may not exist"
+ raise AssertionError(
+ assertion_message
+ or f"No index on '{collection_name}' has keys {dict(expected)}. It has: {present}."
+ )
+
+ self._retry_until_no_assertion_error(check, retry_timeout, retry_pause)
+
@keyword
def document_should_exist(
self,
diff --git a/MongoDBLibraryKeywords.html b/MongoDBLibraryKeywords.html
index f96181d..100306b 100644
--- a/MongoDBLibraryKeywords.html
+++ b/MongoDBLibraryKeywords.html
@@ -6,7 +6,7 @@
diff --git a/README.md b/README.md
index 0381572..3347f14 100644
--- a/README.md
+++ b/README.md
@@ -18,6 +18,7 @@ every keyword, its arguments and examples.
- [Resetting Between Tests](#resetting-between-tests)
- [Waiting For Data](#waiting-for-data)
- [Document Ids](#document-ids)
+- [Diagnosing An Empty Result](#diagnosing-an-empty-result)
- [Connecting To A Hosted Cluster (MongoDB Atlas)](#connecting-to-a-hosted-cluster-mongodb-atlas)
- [Using With AWS](#using-with-aws)
- [Beyond These Keywords](#beyond-these-keywords)
@@ -27,12 +28,16 @@ every keyword, its arguments and examples.
- Connect to a single host, a connection string, or a hosted cluster such as MongoDB Atlas
- Named connections with a connection pool, and clients shared between aliases
- CRUD on one or many documents, with MongoDB query and update operators
+- Seed documents read from Extended JSON files: template placeholders filled from the
+ call's arguments, suite variables substituted, and any field overridable by its path
- Queries with projection, sorting, limiting, skipping and distinct values
- Upserting and whole-document replacement, so a fixture step can run twice
- Collection and database management: create, drop, list
- Index creation, listing and dropping, including unique, sparse and TTL indexes
- Retrying assertions on a query result, a document count, a set of values, or the
- existence of a document, a collection or an index
+ existence of a document, a collection or an index — by its fields as well as its name
+- `Explain Query` for the query that returns nothing and reports no error: the plan the
+ server chose, and the values it actually searched for
- `Run Database Command` for everything the keywords do not wrap
- Runs on Robot Framework 5.0 through 7.x, from one code path
@@ -72,17 +77,29 @@ is `robotframework-mongodblibrary`.
Library MongoDBLibrary coerce_object_ids=${True}
```
-`coerce_object_ids` (default `${True}`) is the library's only import-time argument. It
-controls whether a string `_id` in a query is rewritten to a BSON `ObjectId`; see
-[Document Ids](#document-ids) for what that means and when to turn it off. Everything
-else — hosts, credentials, TLS, auth mechanism — is configured per connection, on the
-connect keywords.
+There are two import-time arguments, and everything else — hosts, credentials, TLS, auth
+mechanism — is configured per connection, on the connect keywords.
+
+`coerce_object_ids` (default `${True}`) controls whether a string `_id` in a query is
+rewritten to a BSON `ObjectId`; see [Document Ids](#document-ids) for what that means and
+when to turn it off.
+
+`document_path` (unset by default) is the directory that documents given to
+`Load Document` and `Insert Document From File` by file name are looked up in; see
+[Documents From Files](#documents-from-files). It removes the repetition of naming the
+directory in every call and does nothing else, so a path given to the keyword still works
+without it.
+
+```robotframework
+*** Settings ***
+Library MongoDBLibrary document_path=${CURDIR}/documents
+```
The library's scope is `GLOBAL`, so one instance is shared by every suite in a run and a
connection opened in one suite is still open in the next. One consequence is worth
knowing: Robot Framework creates a separate instance per set of import arguments, so two
-suites that import with *different* `coerce_object_ids` values get separate instances,
-and therefore separate connection pools rather than shared connections.
+suites that import with *different* argument values get separate instances, and therefore
+separate connection pools rather than shared connections.
## Usage Example
@@ -220,6 +237,176 @@ Query An Id Explicitly
Full details, including exactly what is and is not rewritten, are in the `Object Ids`
section of the [keyword documentation](https://mobynl.github.io/robotframework-mongodblibrary/).
+## Diagnosing An Empty Result
+
+An id that does not match is one cause of a query that finds nothing and errors on
+nothing. `Explain Query` covers the rest: it asks the server how it answered the query,
+and the field to read first is `index_bounds` — the values it actually searched for.
+
+```robotframework
+*** Test Cases ***
+Find Out Why The Document Is Missing
+ ${plan} Explain Query collection_name=readings _id.deviceId=${device_id} _id.date=${date}
+ Log ${plan.index_bounds}
+```
+
+```
+'_id.deviceId': ['["device-1", "device-1"]']
+'_id.date': ['[new Date(1767830400000), new Date(1767830400000)]']
+```
+
+Comparing that with what the suite passed is usually the whole diagnosis. The keyword
+asserts nothing and is not meant to stay in a passing test: put it beside the find that
+returned nothing, read the log, take it out again.
+
+### A compound `_id`
+
+A collection keyed by a subdocument rather than a single value hits three of these at
+once, and none of them errors:
+
+```
+{"_id": {"deviceId": "device-1", "date": ISODate("2026-01-08T00:00:00.001Z")}}
+```
+
+1. **The automatic `_id_` index cannot answer a query on part of the id.** It stores the
+ subdocument as one opaque value, so `_id.deviceId` and `_id.date` are served by a
+ separate index if one exists, and by reading every document if not. Nothing in the
+ suite says that index is load-bearing, so assert it:
+
+ ```robotframework
+ Collection Should Have Index collection_name=readings keys={"_id.deviceId": 1, "_id.date": 1}
+ ```
+
+2. **Matching the whole `_id` is field-order sensitive.** It compares the stored BSON, so
+ the order of the fields is part of the value:
+
+ ```robotframework
+ query={"_id": {"deviceId": "device-1", "date": ${date}}} # matches
+ query={"_id": {"date": ${date}, "deviceId": "device-1"}} # matches nothing, silently
+ ```
+
+3. **A date compares exactly.** A `datetime` at midnight does not match a document stored
+ with milliseconds — which is exactly what the `index_bounds` above make visible.
+
+`Explain Query` also reports `collection_scan`, and it is deliberately *information*
+rather than an assertion. MongoDB rightly chooses a collection scan on a small collection,
+where reading it beats an index lookup plus a fetch, so "this query must not scan" passes
+against production-sized data and fails against a freshly seeded test collection with
+nothing wrong. Where a suite needs an index, `Collection Should Have Index` says so
+directly and cannot flake.
+
+## Documents From Files
+
+A fixture document written into a suite is fine until a second test needs it, and then it
+is copied and the copies drift. `Load Document` reads one from a JSON file, and
+`Insert Document From File` reads it and inserts it in one step:
+
+```robotframework
+*** Settings ***
+Library MongoDBLibrary document_path=${CURDIR}/documents
+
+*** Test Cases ***
+Seed An Order
+ ${document} Load Document order.json
+ ${doc_id} Insert Document From File collection_name=orders path=order.json
+```
+
+The file is MongoDB Extended JSON, so it can hold the types MongoDB stores rather than
+only the ones JSON has syntax for:
+
+```json
+{
+ "_id": {"$oid": "6a7ccdea6abf6a4ebbc3514f"},
+ "placedAt": {"$date": "2026-03-01T09:30:00Z"},
+ "quantity": {"$numberInt": "3"},
+ "total": {"$numberDouble": "42.50"},
+ "email": "${EMAIL}",
+ "lines": [{"sku": "A-1", "quantity": 2}]
+}
+```
+
+That matters for the same reason [Document Ids](#document-ids) does: a string that looks
+like an id does not match one, and a date written as text is stored as text and does not
+compare as a date. Plain JSON values keep their own types. MongoDB's update operators are
+`$`-prefixed too and are passed through untouched, nested values included, so a file can
+hold `{"$set": ..., "$push": ...}` for an update as readily as a document to insert.
+
+`${...}` in the file is replaced from the variables the calling suite can see, and one that
+resolves to nothing fails the keyword naming the file and the variable — rather than
+inserting a document that still says `${EMAIL}` and failing a test somewhere later against
+data that looks almost right.
+
+### Filling A Template
+
+A value that differs on *every* call belongs in the call rather than in a suite variable.
+A file can declare a hole for one, written `{name}` and filled from the keyword's named
+arguments:
+
+```json
+{
+ "unique_id": "{unique_id}",
+ "customerId": "{customerId}",
+ "placedAt": "{placed_at}",
+ "quantity": "{quantity}",
+ "reference": "REF-{unique_id}",
+ "status": "new"
+}
+```
+
+```robotframework
+*** Test Cases ***
+Seed An Order Per Call
+ ${doc_id} Insert Document From File collection_name=orders path=order.json
+ ... unique_id=order-1 customerId=${oid} placed_at=${now} quantity=3
+ ${document} Load Document order.json &{placeholders}
+```
+
+The template stays valid JSON, so editors, `jq` and formatters still read it. Two kinds of
+hole, and the syntax says which is which: `${name}` comes from the suite, `{name}` from the
+call.
+
+A string that is *exactly* one placeholder is replaced whole, quotes included, by the
+value's own Extended JSON form — which is what lets a valid-JSON template carry a value JSON
+cannot write, with no `$oid` or `$date` wrapper needed:
+
+| In the file | Given | Stored as |
+|---|---|---|
+| `"customerId": "{customerId}"` | an ObjectId | a real `ObjectId` |
+| `"placedAt": "{placed_at}"` | a datetime | a real `datetime` |
+| `"quantity": "{quantity}"` | `3` | a real `int` |
+| `"reference": "REF-{unique_id}"` | `order-1` | `"REF-order-1"` |
+
+Inside a string, `{{` and `}}` are literal braces as in `str.format`; JSON's own braces are
+never touched. A hole the file declares that no argument fills is an error naming the file
+and the holes, for the same reason an unresolved `${...}` is.
+
+### Overriding Fields
+
+A field the file already fills can be changed without declaring a hole for it, which is
+what a value that varies only *occasionally* wants — the file's own value stays as the
+default for every test that does not mention it. Any field can be overridden by its path,
+with list positions written as numbers:
+
+```robotframework
+*** Test Cases ***
+Seed Two Orders From One File
+ ${shipped} Load Document order.json status=shipped lines.0.quantity=3
+ ${mine} Load Document order.json customer._id=${customer_id}
+```
+
+This is the part a suite cannot do for itself: `&{dict}` expansion merges one level deep,
+so overriding a nested field otherwise means rebuilding every level above it. A value
+written literally is read the way the file's own values are — `0.8` is a number, `true` is
+a boolean, `{"$oid": "..."}` is an ObjectId, and a word such as `shipped` is text. Every
+step of a path has to exist in the document already; one that does not fails with what the
+document held at that point, because a path that misses is a typo far more often than it is
+a field meant to be added.
+
+Placeholders and overrides are given the same way and the file decides which an argument
+is: a name it declares as a placeholder fills that placeholder, anything else is a path. A
+bare name that is neither fails naming both, since which was meant decides whether the fix
+belongs in the file or in the call.
+
## Connecting To A Hosted Cluster (MongoDB Atlas)
A hosted cluster's name is a DNS seed list rather than a single host, so it needs
diff --git a/atest/document_file_tests.robot b/atest/document_file_tests.robot
new file mode 100644
index 0000000..b93b22a
--- /dev/null
+++ b/atest/document_file_tests.robot
@@ -0,0 +1,223 @@
+*** Settings ***
+Documentation Acceptance tests for reading documents from files: Load Document and
+... Insert Document From File.
+...
+... What needs a real server here is what the types do once they are
+... stored. A document read from a file holds an ObjectId, a datetime, an
+... int and a double, and the point of reading them as BSON types rather
+... than as text is that MongoDB then matches and compares them as such.
+... Only a real server can be asked that: it is the server that answers a
+... query for a date range or an id, and mongomock reimplements those
+... answers rather than giving them.
+...
+... Substitution runs here for a different reason. A variable written in a
+... file is read from the ones the calling suite can see, and that scope
+... exists only while Robot Framework is running a suite: the unit tests
+... substitute against a variable scope they build themselves, and this is
+... where the real one is. The same goes for filling a template's
+... placeholders from named arguments, and from a dictionary expanded into
+... them.
+
+Library Collections
+Library DateTime
+Library MongoDBLibrary document_path=${CURDIR}/documents
+Resource local.resource
+
+Suite Setup Connect To Test Database
+Suite Teardown Cleanup Test Data
+Test Setup Delete All Documents From Collection collection_name=${COLLECTION}
+
+
+*** Variables ***
+${COLLECTION} test_collection_document_file
+${CUSTOMER_ID} 6a7ccdea6abf6a4ebbc3514f
+${ORDER_REFERENCE} REF-1
+
+
+*** Test Cases ***
+Verify A Document Read From A File Is Stored With Its Extended JSON Types
+ [Documentation] The reason the file is Extended JSON: the id is queried as an id
+ ... and the date compares as a date, neither of which text does.
+ Insert Document From File collection_name=${COLLECTION} path=order.json
+
+ ${customer_id} Convert To Object Id ${CUSTOMER_ID}
+ ${found} Find Document collection_name=${COLLECTION} customerId=${customer_id}
+ Should Be Equal ${found.unique_id} test_document_file
+
+ ${from_date} Evaluate {"placedAt": {"$gte": datetime.datetime(2026, 2, 1)}}
+ ${in_range} Count Documents With Query collection_name=${COLLECTION} query=${from_date}
+ Should Be Equal As Integers ${in_range} 1
+
+ # The same date as text matches nothing, which is what a file that stored it as text
+ # would have produced.
+ ${as_text} Count Documents With Query
+ ... collection_name=${COLLECTION}
+ ... query={"placedAt": "2026-03-01T09:30:00Z"}
+ Should Be Equal As Integers ${as_text} 0
+
+Verify A Variable In A File Is Read From The Calling Suite
+ [Documentation] ${ORDER_REFERENCE} in the file resolves to what this suite defines.
+ ${document} Load Document order.json
+ Should Be Equal ${document.reference} REF-1
+
+ ${overridden} Load Document order.json reference=REF-2
+ Should Be Equal ${overridden.reference} REF-2
+
+Verify Overrides Reach A Nested Field And A List Position
+ [Documentation] The one thing a suite cannot do for itself: dictionary expansion
+ ... merges one level deep, so a nested field means rebuilding every level above it.
+ ${document} Load Document order.json
+ ... status=shipped
+ ... customer.address.city=Amsterdam
+ ... lines.0.quantity=5
+ Should Be Equal ${document.status} shipped
+ Should Be Equal ${document.customer.address.city} Amsterdam
+ Should Be Equal ${document.customer.name} A
+ Should Be Equal As Integers ${document.lines}[0][quantity] 5
+ Should Be Equal As Integers ${document.lines}[1][quantity] 1
+
+Verify An Override Is Stored As The Type It Looks Like
+ [Documentation] A number written literally is stored as a number, so it compares as one.
+ Insert Document From File collection_name=${COLLECTION} path=order.json total=99.95
+
+ ${count} Count Documents With Query
+ ... collection_name=${COLLECTION}
+ ... query={"total": {"$gt": 99}}
+ Should Be Equal As Integers ${count} 1
+
+Verify An Update Document Can Be Read From A File
+ [Documentation] Update operators are $-prefixed too, and are passed through as they are.
+ Insert Document From File collection_name=${COLLECTION} path=order.json
+
+ ${update} Load Document ship.json $set.status=shipped
+ Update Document With Operators
+ ... collection_name=${COLLECTION}
+ ... query={"unique_id": "test_document_file"}
+ ... update=${update}
+
+ ${found} Find Document collection_name=${COLLECTION} unique_id=test_document_file
+ Should Be Equal ${found.status} shipped
+ Should Contain ${found.events} shipped
+ ${shipped} Count Documents With Query
+ ... collection_name=${COLLECTION}
+ ... query={"shippedAt": {"$type": "date"}}
+ Should Be Equal As Integers ${shipped} 1
+
+Verify A Template Seeds A Different Document On Every Call
+ [Documentation] The reason placeholders exist: one file, a document per call, with no
+ ... dictionary built in the suite and no suite variable per value.
+ ${customer_id} Convert To Object Id ${CUSTOMER_ID}
+ ${placed_at} Convert Date 2026-03-01 09:30:00 datetime
+
+ FOR ${unique_id} IN dynamic_1 dynamic_2
+ Insert Document From File
+ ... collection_name=${COLLECTION}
+ ... path=dynamic_order.json
+ ... unique_id=${unique_id}
+ ... customerId=${customer_id}
+ ... placed_at=${placed_at}
+ ... quantity=3
+ END
+
+ ${both} Count Documents collection_name=${COLLECTION} customerId=${customer_id}
+ Should Be Equal As Integers ${both} 2
+
+ ${found} Find Document collection_name=${COLLECTION} unique_id=dynamic_2
+ Should Be Equal ${found.reference} REF-dynamic_2
+
+Verify A Filled Placeholder Is Stored As The Type Of The Value Given
+ [Documentation] What the whole-string rule buys: the template stays valid JSON and
+ ... still carries an ObjectId, a date and a number rather than text.
+ ${customer_id} Convert To Object Id ${CUSTOMER_ID}
+ ${placed_at} Convert Date 2026-03-01 09:30:00 datetime
+ Insert Document From File
+ ... collection_name=${COLLECTION}
+ ... path=dynamic_order.json
+ ... unique_id=dynamic_types
+ ... customerId=${customer_id}
+ ... placed_at=${placed_at}
+ ... quantity=3
+
+ ${typed} Evaluate
+ ... {"customerId": {"$type": "objectId"}, "placedAt": {"$type": "date"}, "quantity": {"$type": "int"}}
+ ${count} Count Documents With Query collection_name=${COLLECTION} query=${typed}
+ Should Be Equal As Integers ${count} 1
+
+ ${in_range} Evaluate {"placedAt": {"$gte": datetime.datetime(2026, 2, 1)}}
+ ${matched} Count Documents With Query collection_name=${COLLECTION} query=${in_range}
+ Should Be Equal As Integers ${matched} 1
+
+Verify A Dictionary Can Be Expanded Into The Placeholder Values
+ [Documentation] Robot Framework maps a dictionary into named arguments, so a suite
+ ... that already holds one passes it as it is. An empty one fills nothing.
+ VAR &{placeholders}
+ ... unique_id=dynamic_expanded
+ ... customerId=${CUSTOMER_ID}
+ ... placed_at=2026-03-01T09:30:00Z
+ ... quantity=1
+ ${document} Load Document dynamic_order.json &{placeholders}
+ Should Be Equal ${document.unique_id} dynamic_expanded
+
+ ${unchanged} Load Document order.json &{EMPTY}
+ Should Be Equal ${unchanged.status} new
+
+Verify Placeholders And Overrides Can Be Given In One Call
+ [Documentation] A hole the file declares is filled; anything else is a path into it.
+ ${document} Load Document dynamic_order.json
+ ... unique_id=dynamic_mixed
+ ... customerId=${CUSTOMER_ID}
+ ... placed_at=2026-03-01T09:30:00Z
+ ... quantity=1
+ ... status=shipped
+ ... lines.0.quantity=9
+ Should Be Equal ${document.unique_id} dynamic_mixed
+ Should Be Equal ${document.status} shipped
+ Should Be Equal As Integers ${document.lines}[0][quantity] 9
+
+Verify An Unfilled Placeholder Fails Instead Of Being Stored As Text
+ [Documentation] The literal text a placeholder is written as inserts perfectly well,
+ ... so leaving it would seed a document that looks almost right.
+ Run Keyword And Expect Error *dynamic_order.json*declares*customerId*quantity*
+ ... Load Document dynamic_order.json unique_id=dynamic_unfilled placed_at=x
+ ${count} Count Documents collection_name=${COLLECTION}
+ Should Be Equal As Integers ${count} 0
+
+Verify An Argument That Is Neither A Placeholder Nor A Field Names Both
+ [Documentation] Which one was meant decides whether the fix is in the file or the call.
+ Run Keyword And Expect Error *'stauts' is neither*Placeholders*Fields*
+ ... Load Document dynamic_order.json
+ ... unique_id=dynamic_typo
+ ... customerId=${CUSTOMER_ID}
+ ... placed_at=2026-03-01T09:30:00Z
+ ... quantity=1
+ ... stauts=shipped
+
+Verify An Override Path That Is Not In The Document Fails
+ [Documentation] A path that misses is a typo, so it fails with what was there instead
+ ... of quietly adding a field alongside the one that was meant.
+ Run Keyword And Expect Error *'nmae' is not in 'customer'*Available: name, address*
+ ... Load Document order.json customer.nmae=A
+ Run Keyword And Expect Error *'lines' holds 2 items*index 9 is out of range*
+ ... Load Document order.json lines.9.quantity=1
+
+Verify A Missing Document File Names Where It Looked
+ [Documentation] Reported here rather than as a failure to parse nothing.
+ Run Keyword And Expect Error *missing.json*${CURDIR}${/}documents*
+ ... Load Document missing.json
+
+
+*** Keywords ***
+Connect To Test Database
+ [Documentation] Connect using a host and credentials.
+ Connect To Database
+ ... db_name=${DB_NAME}
+ ... db_user=${DB_USER}
+ ... db_password=${DB_PASSWORD}
+ ... db_host=${DB_HOST}
+ ... db_port=${DB_PORT}
+ ... srv=${DB_SRV}
+
+Cleanup Test Data
+ [Documentation] Remove the collection this suite wrote to, then release the connection.
+ Drop Collection collection_name=${COLLECTION}
+ Disconnect From All Databases
diff --git a/atest/documents/dynamic_order.json b/atest/documents/dynamic_order.json
new file mode 100644
index 0000000..86f2b2b
--- /dev/null
+++ b/atest/documents/dynamic_order.json
@@ -0,0 +1,11 @@
+{
+ "unique_id": "{unique_id}",
+ "customerId": "{customerId}",
+ "placedAt": "{placed_at}",
+ "quantity": "{quantity}",
+ "reference": "REF-{unique_id}",
+ "status": "new",
+ "lines": [
+ {"sku": "A-1", "quantity": 2}
+ ]
+}
diff --git a/atest/documents/order.json b/atest/documents/order.json
new file mode 100644
index 0000000..8d1a3f7
--- /dev/null
+++ b/atest/documents/order.json
@@ -0,0 +1,14 @@
+{
+ "unique_id": "test_document_file",
+ "customerId": {"$oid": "6a7ccdea6abf6a4ebbc3514f"},
+ "placedAt": {"$date": "2026-03-01T09:30:00Z"},
+ "quantity": {"$numberInt": "3"},
+ "total": {"$numberDouble": "42.50"},
+ "status": "new",
+ "reference": "${ORDER_REFERENCE}",
+ "customer": {"name": "A", "address": {"city": "Utrecht"}},
+ "lines": [
+ {"sku": "A-1", "quantity": 2},
+ {"sku": "B-2", "quantity": 1}
+ ]
+}
diff --git a/atest/documents/ship.json b/atest/documents/ship.json
new file mode 100644
index 0000000..8c9d89c
--- /dev/null
+++ b/atest/documents/ship.json
@@ -0,0 +1,4 @@
+{
+ "$set": {"status": "new", "shippedAt": {"$date": "2026-03-02T00:00:00Z"}},
+ "$push": {"events": "shipped"}
+}
diff --git a/atest/explain_tests.robot b/atest/explain_tests.robot
new file mode 100644
index 0000000..728bfb9
--- /dev/null
+++ b/atest/explain_tests.robot
@@ -0,0 +1,164 @@
+*** Settings ***
+Documentation Acceptance tests for `Explain Query` and `Collection Should Have Index`.
+...
+... These need a real server more than any other suite here. What the
+... explain keyword contributes is reading a plan document that only
+... MongoDB produces — mongomock has no ``explain`` at all — and the
+... collection this suite builds is the shape that motivated both
+... keywords: an ``_id`` that is a compound subdocument, queried by
+... dotted path.
+
+Library Collections
+Library MongoDBLibrary
+Resource local.resource
+
+Suite Setup Connect And Seed The Readings Collection
+Suite Teardown Cleanup Test Data
+
+
+*** Variables ***
+${COLLECTION} test_collection_explain
+${INDEX_NAME} _id.deviceId_1__id.date_1
+
+
+*** Test Cases ***
+Verify Explain Query Reports The Index A Dotted Query Uses
+ [Documentation] The motivating query. A dotted path cannot use the automatic
+ ... ``_id_`` index, which stores the subdocument as one opaque value, so this is
+ ... answered by the secondary index the setup creates and by nothing else.
+ ${plan} Explain The Dotted Query
+ Should Be Equal ${plan.collection_scan} ${False}
+ Should Be Equal ${plan.index_name} ${INDEX_NAME}
+ Should Be Equal As Integers ${plan.returned} 1
+ Should Be Equal As Integers ${plan.docs_examined} 1
+
+Verify Explain Query Reports The Values It Searched For
+ [Documentation] The field the keyword exists for: what the server looked for, as it
+ ... understood it. A query matching nothing without erroring is diagnosed here.
+ ${plan} Explain The Dotted Query
+ Dictionary Should Contain Key ${plan.index_bounds} _id.date
+ Should Not Be Empty ${plan.index_bounds}[_id.date]
+
+Verify Explain Query Flags A Collection Scan
+ [Documentation] A field no index covers is read by scanning the collection. Reported
+ ... as information only — the library deliberately has no assertion for it, because
+ ... MongoDB rightly chooses a scan on small collections.
+ ${plan} Explain Query collection_name=${COLLECTION} reading=${3}
+ Should Be Equal ${plan.collection_scan} ${True}
+ Should Be Equal ${plan.index_name} ${None}
+ Should Be True ${plan.docs_examined} > 1
+
+Verify Explain Query Shows Why A Reversed Id Matches Nothing
+ [Documentation] Subdocument equality compares the BSON as stored, so the field order
+ ... is part of the value. Reversed, the query matches nothing and reports no error;
+ ... the explain says it searched and returned none, which is the visible difference.
+ ${date} Get The Seeded Date
+ ${matching} Explain Query
+ ... collection_name=${COLLECTION}
+ ... query=${{ {"_id": {"deviceId": "device-1", "date": $date}} }}
+ ${reversed} Explain Query
+ ... collection_name=${COLLECTION}
+ ... query=${{ {"_id": {"date": $date, "deviceId": "device-1"}} }}
+ Should Be Equal As Integers ${matching.returned} 1
+ Should Be Equal As Integers ${reversed.returned} 0
+ ${document} Find Document With Query
+ ... collection_name=${COLLECTION}
+ ... query=${{ {"_id": {"date": $date, "deviceId": "device-1"}} }}
+ Should Be Equal ${document} ${None}
+
+Verify Explain Query Takes The Query Either Way
+ [Documentation] Both find keywords' argument shapes explain the same query, so an
+ ... explain can be dropped in beside either without rewriting the call.
+ ${date} Get The Seeded Date
+ ${by_params} Explain Query
+ ... collection_name=${COLLECTION}
+ ... _id.deviceId=device-1
+ ... _id.date=${date}
+ ${by_query} Explain Query
+ ... collection_name=${COLLECTION}
+ ... query=${{ {"_id.deviceId": "device-1", "_id.date": $date} }}
+ Should Be Equal ${by_params.index_name} ${by_query.index_name}
+ Should Be Equal As Integers ${by_params.returned} ${by_query.returned}
+
+Verify Explain Query Refuses Both Query Forms At Once
+ [Documentation] Merging them would hide a typo, and which was meant decides the fix.
+ Run Keyword And Expect Error *not both*
+ ... Explain Query collection_name=${COLLECTION} query={"reading": 3} reading=${3}
+
+Verify Explain Query Can Plan Without Running The Query
+ [Documentation] ``queryPlanner`` verbosity picks a plan and stops, so the counters
+ ... are empty rather than zero. Not an error: nothing was executed to count.
+ ${plan} Explain Query
+ ... collection_name=${COLLECTION}
+ ... reading=${3}
+ ... verbosity=queryPlanner
+ Should Be Equal ${plan.collection_scan} ${True}
+ Should Be Equal ${plan.keys_examined} ${None}
+ Should Be Equal ${plan.docs_examined} ${None}
+ Should Be Equal ${plan.returned} ${None}
+
+Verify Explain Query Returns The Server's Own Explain
+ [Documentation] The escape hatch: anything the summary leaves out is still reachable.
+ ${plan} Explain The Dotted Query
+ Dictionary Should Contain Key ${plan.raw} queryPlanner
+ Dictionary Should Contain Key ${plan.raw} executionStats
+
+Verify Collection Should Have Index Passes For The Index That Exists
+ [Documentation] Asked by fields, so the derived name never has to be written out.
+ Collection Should Have Index
+ ... collection_name=${COLLECTION}
+ ... keys={"_id.deviceId": 1, "_id.date": 1}
+
+Verify Collection Should Have Index Reports Which Indexes Are There
+ [Documentation] The failure has to say what the collection does have, or the next
+ ... step is another round of looking.
+ Run Keyword And Expect Error *No index on '${COLLECTION}' has keys*It has:*${INDEX_NAME}*
+ ... Collection Should Have Index collection_name=${COLLECTION} keys={"status": 1}
+
+Verify Collection Should Have Index Is Field Order Sensitive
+ [Documentation] A compound index serves its fields left to right, so the same fields
+ ... in the other order are a different index and cannot answer the same queries.
+ Run Keyword And Expect Error *No index on '${COLLECTION}' has keys*
+ ... Collection Should Have Index collection_name=${COLLECTION} keys={"_id.date": 1, "_id.deviceId": 1}
+
+
+*** Keywords ***
+Cleanup Test Data
+ [Documentation] Remove the collection this suite created, then release the connection.
+ Drop Collection collection_name=${COLLECTION}
+ Disconnect From All Databases
+
+Connect And Seed The Readings Collection
+ [Documentation] Connect, then build a collection keyed by a compound ``_id``.
+ ...
+ ... Two hundred documents, which is enough for a collection scan to read visibly
+ ... more than an index lookup does. The secondary index is the load-bearing one:
+ ... without it every dotted query below scans.
+ Connect To Database
+ ... db_name=${DB_NAME}
+ ... db_user=${DB_USER}
+ ... db_password=${DB_PASSWORD}
+ ... db_host=${DB_HOST}
+ ... db_port=${DB_PORT}
+ ... srv=${DB_SRV}
+ Drop Collection collection_name=${COLLECTION}
+ ${start} Evaluate datetime.datetime(2026, 1, 1) modules=datetime
+ ${day} Evaluate datetime.timedelta(days=1) modules=datetime
+ ${documents} Evaluate
+ ... [{"_id": {"deviceId": f"device-{u}", "date": $start + $day * d}, "reading": u + d} for u in range(20) for d in range(10)]
+ Insert Documents collection_name=${COLLECTION} documents=${documents}
+ Create Index collection_name=${COLLECTION} keys={"_id.deviceId": 1, "_id.date": 1}
+
+Explain The Dotted Query
+ [Documentation] The query that motivated these keywords, explained.
+ ${date} Get The Seeded Date
+ ${plan} Explain Query
+ ... collection_name=${COLLECTION}
+ ... _id.deviceId=device-1
+ ... _id.date=${date}
+ RETURN ${plan}
+
+Get The Seeded Date
+ [Documentation] The date every ``device-1`` assertion below matches on.
+ ${date} Evaluate datetime.datetime(2026, 1, 8) modules=datetime
+ RETURN ${date}
diff --git a/pyproject.toml b/pyproject.toml
index cfd04db..2d8d383 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "robotframework-mongodb"
-version = "1.1.0"
+version = "1.2.0"
description = "MongoDB test library for Robot Framework."
authors = ["MobyNl "]
readme = "README.md"
diff --git a/utest/test_explain_query.py b/utest/test_explain_query.py
new file mode 100644
index 0000000..41ec949
--- /dev/null
+++ b/utest/test_explain_query.py
@@ -0,0 +1,448 @@
+"""Unit tests for `Explain Query`.
+
+These run against the ``mock_db`` MagicMock rather than mongomock, which implements no
+``explain`` at all. That is the right tool here anyway: what the keyword contributes is
+reading a plan tree whose shape differs by server version and topology, and the only way
+to cover those shapes at once is to feed it the documents each of them produces. The
+acceptance suite covers a real server answering a real query.
+"""
+
+import pytest
+from bson import ObjectId
+
+BOUNDS = {"_id.date": ["[new Date(1702996077710), new Date(1702996077710)]"]}
+
+
+def classic_explain():
+ """A pre-SBE plan: the winning plan's stage sits at the top of the tree."""
+ return {
+ "queryPlanner": {
+ "winningPlan": {
+ "stage": "FETCH",
+ "inputStage": {
+ "stage": "IXSCAN",
+ "indexName": "_id.deviceId_1__id.date_1",
+ "indexBounds": BOUNDS,
+ },
+ }
+ },
+ "executionStats": {
+ "totalKeysExamined": 1,
+ "totalDocsExamined": 1,
+ "nReturned": 1,
+ "executionTimeMillis": 0,
+ },
+ }
+
+
+def sbe_explain():
+ """A slot-based-engine plan, which nests what the classic one puts at the top."""
+ return {
+ "queryPlanner": {
+ "winningPlan": {
+ "queryPlan": {
+ "stage": "FETCH",
+ "inputStage": {
+ "stage": "IXSCAN",
+ "indexName": "_id.deviceId_1__id.date_1",
+ "indexBounds": BOUNDS,
+ },
+ },
+ "slotBasedPlan": {"slots": "$$RESULT=s11"},
+ }
+ },
+ "executionStats": {
+ "totalKeysExamined": 1,
+ "totalDocsExamined": 1,
+ "nReturned": 1,
+ "executionTimeMillis": 3,
+ },
+ }
+
+
+def collection_scan_explain():
+ return {
+ "queryPlanner": {"winningPlan": {"stage": "COLLSCAN", "direction": "forward"}},
+ "executionStats": {
+ "totalKeysExamined": 0,
+ "totalDocsExamined": 3639,
+ "nReturned": 1,
+ "executionTimeMillis": 3,
+ },
+ }
+
+
+def sharded_explain():
+ return {
+ "queryPlanner": {
+ "winningPlan": {
+ "stage": "SHARD_MERGE",
+ "shards": [
+ {
+ "shardName": "shard-a",
+ "winningPlan": {
+ "stage": "FETCH",
+ "inputStage": {"stage": "IXSCAN", "indexName": "deviceId_1", "indexBounds": BOUNDS},
+ },
+ },
+ {
+ "shardName": "shard-b",
+ "winningPlan": {"stage": "COLLSCAN", "direction": "forward"},
+ },
+ ],
+ }
+ },
+ "executionStats": {
+ "totalKeysExamined": 1,
+ "totalDocsExamined": 2048,
+ "nReturned": 1,
+ "executionTimeMillis": 11,
+ "executionStages": {
+ "stage": "SHARD_MERGE",
+ "shards": [
+ {
+ "shardName": "shard-a",
+ "totalKeysExamined": 1,
+ "totalDocsExamined": 1,
+ "nReturned": 1,
+ # The root stage counts what it alone read, which for a FETCH over
+ # an IXSCAN is not the shard's total. The totals are what to report.
+ "executionStages": {"stage": "FETCH", "docsExamined": 1, "nReturned": 1},
+ },
+ {
+ "shardName": "shard-b",
+ "totalKeysExamined": 0,
+ "totalDocsExamined": 2047,
+ "nReturned": 0,
+ "executionStages": {"stage": "COLLSCAN", "docsExamined": 2047, "nReturned": 0},
+ },
+ ],
+ },
+ },
+ }
+
+
+@pytest.fixture
+def explains(mongo_keywords, mock_db):
+ """Keywords whose explain command returns whatever the test sets as ``returns``."""
+
+ def set_return(explain):
+ mock_db.command.return_value = explain
+ return mongo_keywords
+
+ set_return.command = mock_db.command # type: ignore[attr-defined]
+ return set_return
+
+
+def test_explain_query_summarises_a_classic_plan(explains):
+ mongo = explains(classic_explain())
+
+ plan = mongo.explain_query("readings", **{"_id.deviceId": "device-1"})
+
+ assert plan["stage"] == "FETCH"
+ assert plan["index_name"] == "_id.deviceId_1__id.date_1"
+ assert plan["collection_scan"] is False
+ assert plan["keys_examined"] == 1
+ assert plan["docs_examined"] == 1
+ assert plan["returned"] == 1
+ assert plan["duration_ms"] == 0
+
+
+def test_explain_query_keeps_the_index_bounds(explains):
+ """The field the keyword exists for: what the server actually searched for.
+
+ A query that matches nothing and errors on nothing is answered here and nowhere else,
+ so this is the one field the summary must never drop while flattening.
+ """
+ mongo = explains(classic_explain())
+
+ plan = mongo.explain_query("readings", **{"_id.deviceId": "device-1"})
+
+ assert plan["index_bounds"] == BOUNDS
+
+
+def test_explain_query_reads_a_slot_based_plan(explains):
+ """The stage moved under ``queryPlan`` in the newer engine, and must still be found."""
+ mongo = explains(sbe_explain())
+
+ plan = mongo.explain_query("readings", **{"_id.deviceId": "device-1"})
+
+ assert plan["stage"] == "FETCH"
+ assert plan["index_name"] == "_id.deviceId_1__id.date_1"
+ assert plan["index_bounds"] == BOUNDS
+ assert plan["collection_scan"] is False
+
+
+@pytest.mark.parametrize("stage", ["EXPRESS_IXSCAN", "IDHACK"], ids=["mongodb_8", "older"])
+def test_explain_query_treats_every_id_fast_path_as_an_index(explains, stage):
+ """The ``_id`` fast path is named differently by server version.
+
+ Detecting a scan by the presence of a COLLSCAN rather than by an allowlist of index
+ stage names is what makes this hold without a version check.
+ """
+ mongo = explains(
+ {
+ "queryPlanner": {"winningPlan": {"stage": stage, "indexName": "_id_"}},
+ "executionStats": {
+ "totalKeysExamined": 1,
+ "totalDocsExamined": 1,
+ "nReturned": 1,
+ "executionTimeMillis": 0,
+ },
+ }
+ )
+
+ plan = mongo.explain_query("readings", query={"_id": {"deviceId": "d-1"}})
+
+ assert plan["stage"] == stage
+ assert plan["collection_scan"] is False
+ assert plan["index_name"] == "_id_"
+
+
+def test_explain_query_flags_a_collection_scan(explains):
+ mongo = explains(collection_scan_explain())
+
+ plan = mongo.explain_query("readings", status="new")
+
+ assert plan["collection_scan"] is True
+ assert plan["index_name"] is None
+ assert plan["index_bounds"] is None
+ assert plan["docs_examined"] == 3639
+
+
+def test_explain_query_ignores_a_rejected_collection_scan(explains):
+ """A candidate plan the server threw away describes work that never happened.
+
+ Sorting by a field the index does not cover is enough to put a COLLSCAN among the
+ rejected plans of a query the server answered with an index, so reporting one would
+ send a suite hunting for a scan that is not there.
+ """
+ explain = classic_explain()
+ explain["queryPlanner"]["rejectedPlans"] = [{"stage": "SORT", "inputStage": {"stage": "COLLSCAN"}}]
+ mongo = explains(explain)
+
+ plan = mongo.explain_query("readings", **{"_id.deviceId": "device-1"})
+
+ assert plan["collection_scan"] is False
+ assert plan["index_name"] == "_id.deviceId_1__id.date_1"
+
+
+def test_explain_query_does_not_borrow_an_index_from_a_rejected_plan(explains):
+ """The mirror case, and the worse one: an index reported for a query that scanned."""
+ explain = collection_scan_explain()
+ explain["queryPlanner"]["rejectedPlans"] = [
+ {"stage": "FETCH", "inputStage": {"stage": "IXSCAN", "indexName": "status_1", "indexBounds": BOUNDS}}
+ ]
+ mongo = explains(explain)
+
+ plan = mongo.explain_query("readings", status="new")
+
+ assert plan["collection_scan"] is True
+ assert plan["index_name"] is None
+ assert plan["index_bounds"] is None
+
+
+def test_explain_query_ignores_the_trial_runs_of_the_plans_it_did_not_pick(explains):
+ """``allPlansExecution`` verbosity carries every candidate's trial run as well."""
+ explain = classic_explain()
+ explain["executionStats"]["allPlansExecution"] = [
+ {"executionStages": {"stage": "COLLSCAN", "docsExamined": 3639}}
+ ]
+ mongo = explains(explain)
+
+ assert mongo.explain_query("readings", status="new")["collection_scan"] is False
+
+
+def test_explain_query_ignores_a_shards_rejected_plans(explains):
+ """Each shard chooses its own plan, and keeps its own rejects alongside it."""
+ explain = sharded_explain()
+ shard = explain["queryPlanner"]["winningPlan"]["shards"][0]
+ shard["rejectedPlans"] = [{"stage": "COLLSCAN", "direction": "forward"}]
+ mongo = explains(explain)
+
+ summaries = {summary["shard"]: summary for summary in mongo.explain_query("readings", status="new")["shards"]}
+
+ assert summaries["shard-a"]["collection_scan"] is False
+ assert summaries["shard-b"]["collection_scan"] is True
+
+
+def test_explain_query_summarises_each_shard(explains):
+ mongo = explains(sharded_explain())
+
+ plan = mongo.explain_query("readings", status="new")
+
+ assert plan["stage"] == "SHARD_MERGE"
+ assert plan["docs_examined"] == 2048
+ assert plan["shards"] == [
+ {
+ "shard": "shard-a",
+ "stage": "FETCH",
+ "index_name": "deviceId_1",
+ "index_bounds": BOUNDS,
+ "collection_scan": False,
+ "keys_examined": 1,
+ "docs_examined": 1,
+ "returned": 1,
+ },
+ {
+ "shard": "shard-b",
+ "stage": "COLLSCAN",
+ "index_name": None,
+ "index_bounds": None,
+ "collection_scan": True,
+ "keys_examined": 0,
+ "docs_examined": 2047,
+ "returned": 0,
+ },
+ ]
+
+
+def test_explain_query_leaves_out_shards_when_the_plan_is_not_sharded(explains):
+ mongo = explains(classic_explain())
+
+ assert "shards" not in mongo.explain_query("readings", status="new")
+
+
+def test_explain_query_returns_the_raw_explain(explains):
+ """The escape hatch: a field the summary omits is still reachable."""
+ explain = classic_explain()
+ mongo = explains(explain)
+
+ assert mongo.explain_query("readings", status="new")["raw"] == explain
+
+
+def test_explain_query_logs_the_summary(explains, mocker):
+ """Logged at INFO, never at WARN: a warning on correct use trains people to ignore them."""
+ info = mocker.patch("MongoDBLibrary.keywords.logger.info")
+ warn = mocker.patch("MongoDBLibrary.keywords.logger.warn", create=True)
+ mongo = explains(classic_explain())
+
+ mongo.explain_query("readings", status="new")
+
+ assert "_id.deviceId_1__id.date_1" in info.call_args.args[0]
+ warn.assert_not_called()
+
+
+def test_explain_query_logs_one_field_per_line(explains, mocker):
+ """Robot Framework keeps newlines and indentation in the log, so the summary uses them.
+
+ On one line it is unreadable, and ``index_bounds`` — a dictionary inside the summary,
+ and the field the keyword exists for — suffers most.
+ """
+ info = mocker.patch("MongoDBLibrary.keywords.logger.info")
+ mongo = explains(classic_explain())
+
+ mongo.explain_query("readings", status="new")
+
+ logged = info.call_args.args[0].splitlines()
+ assert logged[0] == "Explain of {'status': 'new'} on 'readings':"
+ assert " stage FETCH" in logged
+ assert " index_bounds" in logged
+ assert " _id.date [new Date(1702996077710), new Date(1702996077710)]" in logged
+
+
+def test_explain_query_logs_the_whole_explain_at_debug(explains, mocker):
+ """Indented, and at DEBUG: too long to read every time, wanted on the occasion it is."""
+ debug = mocker.patch("MongoDBLibrary.keywords.logger.debug")
+ mongo = explains(classic_explain())
+
+ mongo.explain_query("readings", status="new")
+
+ logged = debug.call_args.args[0]
+ assert '\n "queryPlanner": {' in logged
+
+
+def test_explain_query_debug_log_survives_a_value_json_cannot_write(explains, mocker):
+ """An explain holds BSON types, and failing to log must not fail the keyword."""
+ debug = mocker.patch("MongoDBLibrary.keywords.logger.debug")
+ explain = classic_explain()
+ explain["queryPlanner"]["parsedQuery"] = {"_id": ObjectId("6a7ccdea6abf6a4ebbc3514f")}
+ mongo = explains(explain)
+
+ mongo.explain_query("readings", status="new")
+
+ assert "6a7ccdea6abf6a4ebbc3514f" in debug.call_args.args[0]
+
+
+def test_explain_query_without_execution_stats_leaves_the_counters_empty(explains):
+ """``queryPlanner`` verbosity picks a plan without running it, which is not an error."""
+ mongo = explains({"queryPlanner": {"winningPlan": {"stage": "COLLSCAN"}}})
+
+ plan = mongo.explain_query("readings", status="new", verbosity="queryPlanner")
+
+ assert plan["stage"] == "COLLSCAN"
+ assert plan["keys_examined"] is None
+ assert plan["docs_examined"] is None
+ assert plan["returned"] is None
+ assert plan["duration_ms"] is None
+
+
+def test_explain_query_sends_the_query_as_a_find_command(explains):
+ mongo = explains(classic_explain())
+
+ mongo.explain_query("readings", **{"_id.deviceId": "device-1"})
+
+ assert explains.command.call_args.args == (
+ {"explain": {"find": "readings", "filter": {"_id.deviceId": "device-1"}}, "verbosity": "executionStats"},
+ )
+
+
+def test_explain_query_sends_the_rest_of_the_find(explains):
+ mongo = explains(classic_explain())
+
+ mongo.explain_query(
+ "readings",
+ query={"status": "new"},
+ projection={"total": 1},
+ sort={"placedAt": -1},
+ limit=5,
+ skip=2,
+ verbosity="allPlansExecution",
+ )
+
+ assert explains.command.call_args.args == (
+ {
+ "explain": {
+ "find": "readings",
+ "filter": {"status": "new"},
+ "projection": {"total": 1},
+ "sort": {"placedAt": -1},
+ "limit": 5,
+ "skip": 2,
+ },
+ "verbosity": "allPlansExecution",
+ },
+ )
+
+
+def test_explain_query_leaves_out_the_options_it_was_not_given(explains):
+ """A `limit` of zero means no limit, and the find command must not be told otherwise."""
+ mongo = explains(classic_explain())
+
+ mongo.explain_query("readings", status="new")
+
+ assert explains.command.call_args.args[0]["explain"] == {"find": "readings", "filter": {"status": "new"}}
+
+
+def test_explain_query_explains_the_query_that_would_really_be_sent(explains):
+ """A string ``_id`` is rewritten for the find keywords, so it is rewritten here too."""
+ mongo = explains(classic_explain())
+
+ mongo.explain_query("orders", _id="6a7ccdea6abf6a4ebbc3514f")
+
+ assert explains.command.call_args.args[0]["explain"]["filter"] == {
+ "_id": ObjectId("6a7ccdea6abf6a4ebbc3514f")
+ }
+
+
+def test_explain_query_refuses_both_ways_of_giving_the_query(explains):
+ """Merging them would hide a typo, and which one was meant decides what to fix."""
+ mongo = explains(classic_explain())
+
+ with pytest.raises(ValueError, match="not both"):
+ mongo.explain_query("readings", query={"status": "new"}, status="new")
+
+
+def test_explain_query_reports_a_missing_alias_like_every_other_keyword(mongo_keywords):
+ with pytest.raises(KeyError, match="Alias 'missing_alias' not found in connection pool."):
+ mongo_keywords.explain_query("readings", alias="missing_alias", status="new")
diff --git a/utest/test_keywords.py b/utest/test_keywords.py
index ea0dd86..1818cc3 100644
--- a/utest/test_keywords.py
+++ b/utest/test_keywords.py
@@ -920,6 +920,67 @@ def test_check_index_exists_honours_retry_timeout(mongo):
)
+def test_collection_should_have_index(mongo):
+ mongo.insert_document("readings", {"_id": {"deviceId": "d", "date": 1}})
+ mongo.create_index("readings", {"_id.deviceId": 1, "_id.date": 1})
+
+ mongo.collection_should_have_index("readings", {"_id.deviceId": 1, "_id.date": 1})
+
+
+def test_collection_should_have_index_ignores_the_derived_name(mongo):
+ """The point of asking by fields: the same index under a name of someone's choosing."""
+ mongo.insert_document("users", {"email": "a@example.test"})
+ mongo.create_index("users", {"email": 1}, index_name="by_email")
+
+ mongo.collection_should_have_index("users", {"email": 1})
+
+
+def test_collection_should_have_index_reports_what_is_there(mongo):
+ mongo.insert_document("users", {"email": "a@example.test"})
+
+ with pytest.raises(AssertionError, match=r"No index on 'users' has keys.*It has: _id_ \{'_id': 1\}"):
+ mongo.collection_should_have_index("users", {"email": 1})
+
+
+def test_collection_should_have_index_says_when_there_is_no_collection(mongo):
+ """A collection that exists has ``_id_``, so nothing at all is the likelier diagnosis.
+
+ Naming it beats a sentence that trails off after ``It has:``, which is what a typo in
+ the collection name used to produce.
+ """
+ mongo.insert_document("readings", {"email": "a@example.test"})
+
+ with pytest.raises(AssertionError, match="no indexes at all, so the collection may not exist"):
+ mongo.collection_should_have_index("redings", {"email": 1})
+
+
+def test_collection_should_have_index_is_field_order_sensitive(mongo):
+ """A compound index serves its fields left to right, so the order is the index."""
+ mongo.insert_document("readings", {"_id": {"deviceId": "d", "date": 1}})
+ mongo.create_index("readings", {"_id.deviceId": 1, "_id.date": 1})
+
+ with pytest.raises(AssertionError, match="No index on 'readings' has keys"):
+ mongo.collection_should_have_index("readings", {"_id.date": 1, "_id.deviceId": 1})
+
+
+def test_collection_should_have_index_takes_a_custom_message(mongo):
+ mongo.insert_document("users", {"email": "a@example.test"})
+
+ with pytest.raises(AssertionError, match="the login query needs this"):
+ mongo.collection_should_have_index(
+ "users", {"email": 1}, assertion_message="the login query needs this"
+ )
+
+
+def test_collection_should_have_index_honours_retry_timeout(mongo):
+ mongo.insert_document("users", {"email": "a@example.test"})
+
+ with pytest.raises(AssertionError, match="No index on 'users' has keys"):
+ mongo.collection_should_have_index(
+ "users", {"email": 1}, retry_timeout="100 milliseconds", retry_pause="0 seconds"
+ )
+
+
def test_document_should_exist(mongo):
mongo.insert_document("orders", {"order_id": "A-1"})
@@ -1010,6 +1071,7 @@ def test_existence_assertions_keep_their_query_argument_types(keyword_name):
param("check_distinct_values", ("orders", "status", AssertionOperator.equal, []), {}, id="check_distinct"),
param("check_collection_exists", ("orders",), {}, id="check_collection_exists"),
param("check_index_exists", ("orders", "by_email"), {}, id="check_index_exists"),
+ param("collection_should_have_index", ("orders", {"email": 1}), {}, id="collection_should_have_index"),
param("document_should_exist", ("orders",), {"key": "value"}, id="document_should_exist"),
param("document_should_not_exist", ("orders",), {"key": "value"}, id="document_should_not_exist"),
],
diff --git a/utest/test_load_document.py b/utest/test_load_document.py
new file mode 100644
index 0000000..f0c2bf7
--- /dev/null
+++ b/utest/test_load_document.py
@@ -0,0 +1,671 @@
+"""Tests for reading documents from files: Load Document and Insert Document From File."""
+
+import datetime
+import re
+
+import pytest
+from bson import ObjectId
+from pytest import param
+from robot.errors import VariableError
+from robot.libraries.BuiltIn import RobotNotRunningError
+from robot.utils import DotDict
+from robot.variables import Variables
+
+from MongoDBLibrary.keywords import MongoDBKeywords
+
+OID = "6a7ccdea6abf6a4ebbc3514f"
+
+
+@pytest.fixture
+def documents(tmp_path):
+ """A directory to write document files into."""
+ directory = tmp_path / "documents"
+ directory.mkdir()
+ return directory
+
+
+@pytest.fixture
+def write(documents):
+ """Write a document file and return its bare name."""
+
+ def write_document(name, text):
+ (documents / name).write_text(text, encoding="utf-8")
+ return name
+
+ return write_document
+
+
+@pytest.fixture
+def suite_variables(mocker):
+ """Robot Framework's own variable scope, standing in for a running suite.
+
+ Substitution is exercised against `robot.variables.Variables` rather than a stub, so
+ the tests see Robot Framework's real behaviour: how it renders a value inside a larger
+ string, and what it raises for a name that is not there.
+ """
+ variables = Variables()
+ builtin = mocker.patch("MongoDBLibrary.documents.BuiltIn").return_value
+ builtin.replace_variables.side_effect = variables.replace_scalar
+ return variables
+
+
+@pytest.fixture
+def loader(connection_manager, documents):
+ """Keywords that look documents up in the document directory."""
+ return MongoDBKeywords(connection_manager, document_path=str(documents))
+
+
+# --------------------------------------------------------------------------- #
+# Finding the file
+# --------------------------------------------------------------------------- #
+
+def test_a_bare_file_name_is_resolved_against_the_document_path(loader, write):
+ write("order.json", '{"status": "new"}')
+
+ assert loader.load_document("order.json") == {"status": "new"}
+
+
+def test_a_path_still_works_without_a_document_path(connection_manager, documents, write):
+ write("order.json", '{"status": "new"}')
+ keywords = MongoDBKeywords(connection_manager)
+
+ assert keywords.load_document(str(documents / "order.json")) == {"status": "new"}
+
+
+def test_a_path_is_used_as_written_even_when_a_document_path_is_set(loader, tmp_path):
+ elsewhere = tmp_path / "elsewhere.json"
+ elsewhere.write_text('{"status": "elsewhere"}', encoding="utf-8")
+
+ assert loader.load_document(str(elsewhere)) == {"status": "elsewhere"}
+
+
+def test_a_missing_file_reports_where_it_looked(loader, documents):
+ with pytest.raises(ValueError) as error:
+ loader.load_document("missing.json")
+
+ assert "missing.json" in str(error.value)
+ assert str(documents / "missing.json") in str(error.value)
+
+
+# --------------------------------------------------------------------------- #
+# Extended JSON
+# --------------------------------------------------------------------------- #
+
+def test_extended_json_types_become_the_types_mongodb_stores(loader, write):
+ write(
+ "order.json",
+ """
+ {
+ "_id": {"$oid": "%s"},
+ "placedAt": {"$date": "2026-03-01T09:30:00Z"},
+ "quantity": {"$numberInt": "3"},
+ "total": {"$numberDouble": "42.50"}
+ }
+ """ % OID,
+ )
+
+ document = loader.load_document("order.json")
+
+ assert document["_id"] == ObjectId(OID)
+ assert document["placedAt"] == datetime.datetime(2026, 3, 1, 9, 30)
+ assert isinstance(document["quantity"], int) and document["quantity"] == 3
+ assert isinstance(document["total"], float) and document["total"] == 42.50
+
+
+def test_plain_json_values_keep_their_natural_types(loader, write):
+ write("order.json", '{"status": "new", "lines": 2, "total": 1.5, "paid": true, "note": null}')
+
+ document = loader.load_document("order.json")
+
+ assert document == {"status": "new", "lines": 2, "total": 1.5, "paid": True, "note": None}
+ assert isinstance(document["lines"], int)
+ assert isinstance(document["total"], float)
+
+
+def test_update_operators_are_passed_through(loader, write):
+ """A ``$``-prefixed operator is not an extended type, so an update file works too."""
+ write(
+ "ship.json",
+ '{"$set": {"status": "shipped", "shippedAt": {"$date": "2026-03-02T00:00:00Z"}},'
+ ' "$push": {"events": "shipped"}, "$inc": {"revision": 1}}',
+ )
+
+ document = loader.load_document("ship.json")
+
+ assert set(document) == {"$set", "$push", "$inc"}
+ assert document["$set"]["status"] == "shipped"
+ assert document["$set"]["shippedAt"] == datetime.datetime(2026, 3, 2, 0, 0)
+ assert document["$push"] == {"events": "shipped"}
+ assert document["$inc"] == {"revision": 1}
+
+
+def test_invalid_json_reports_the_file_with_the_line_and_column(loader, write):
+ """Where the position points is Python's business, so only that there is one is asserted.
+
+ Python 3.13 rewrote the decoder's messages: a trailing comma is now reported at the
+ comma itself rather than at the token that followed it, so pinning the numbers would
+ pin the interpreter version instead of the behaviour.
+ """
+ write("order.json", '{\n "status": "new",\n}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json")
+
+ assert "order.json" in str(error.value)
+ assert re.search(r"at line \d+ column \d+\.$", str(error.value))
+
+
+def test_a_value_extended_json_cannot_read_surfaces_its_own_error(loader, write):
+ """A literal ``$date`` holding something that is not a date is json_util's failure."""
+ write("order.json", '{"placedAt": {"$date": "yesterday"}}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json")
+
+ assert "order.json" in str(error.value)
+ assert "yesterday" in str(error.value)
+
+
+def test_a_file_that_does_not_hold_an_object_is_reported(loader, write):
+ write("orders.json", '[{"status": "new"}]')
+
+ with pytest.raises(ValueError, match="orders.json"):
+ loader.load_document("orders.json")
+
+
+# --------------------------------------------------------------------------- #
+# Suite variables, written ${...}
+# --------------------------------------------------------------------------- #
+
+def test_variables_are_replaced_from_the_suite_scope(loader, write, suite_variables):
+ suite_variables["${EMAIL}"] = "a@example.test"
+ suite_variables["${QUANTITY}"] = 3
+ write("order.json", '{"email": "${EMAIL}", "quantity": ${QUANTITY}}')
+
+ assert loader.load_document("order.json") == {"email": "a@example.test", "quantity": 3}
+
+
+def test_a_variable_can_carry_an_extended_json_value(loader, write, suite_variables):
+ suite_variables["${CUSTOMER_ID}"] = OID
+ write("order.json", '{"customerId": {"$oid": "${CUSTOMER_ID}"}}')
+
+ assert loader.load_document("order.json") == {"customerId": ObjectId(OID)}
+
+
+def test_an_undefined_variable_fails_with_the_file_and_the_variable(loader, write, suite_variables):
+ write("order.json", '{"email": "${EMAIL}"}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json")
+
+ assert "order.json" in str(error.value)
+ assert "EMAIL" in str(error.value)
+ assert isinstance(error.value.__cause__, VariableError)
+
+
+def test_a_file_without_variables_needs_no_running_suite(loader, write):
+ """Nothing to substitute, so nothing asks Robot Framework for a variable scope."""
+ write("order.json", '{"status": "new"}')
+
+ assert loader.load_document("order.json") == {"status": "new"}
+
+
+def test_a_variable_outside_a_running_suite_is_reported_as_such(loader, write):
+ write("order.json", '{"email": "${EMAIL}"}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json")
+
+ assert "order.json" in str(error.value)
+ assert isinstance(error.value.__cause__, RobotNotRunningError)
+
+
+def test_a_whole_file_variable_resolves_to_the_variable_itself(loader, write, suite_variables):
+ suite_variables["${ORDER}"] = {"status": "new", "lines": [{"quantity": 1}]}
+ write("order.json", "${ORDER}")
+
+ assert loader.load_document("order.json") == {"status": "new", "lines": [{"quantity": 1}]}
+
+
+def test_overriding_a_whole_file_variable_leaves_the_variable_alone(loader, write, suite_variables):
+ original = {"status": "new", "lines": [{"quantity": 1}]}
+ suite_variables["${ORDER}"] = original
+ write("order.json", "${ORDER}")
+
+ loader.load_document("order.json", **{"lines.0.quantity": "5"})
+
+ assert original == {"status": "new", "lines": [{"quantity": 1}]}
+
+
+# --------------------------------------------------------------------------- #
+# Placeholders, written {...} and filled from the keyword's arguments
+# --------------------------------------------------------------------------- #
+
+@pytest.mark.parametrize(
+ "given, expected",
+ [
+ param("Mark", "Mark", id="text"),
+ param("3", 3, id="a whole number written as text"),
+ param("0.8", 0.8, id="a number written as text"),
+ param(3, 3, id="an int"),
+ param(True, True, id="a boolean"),
+ param(None, None, id="none"),
+ param(ObjectId(OID), ObjectId(OID), id="an object id"),
+ param(datetime.datetime(2026, 3, 1, 9, 30), datetime.datetime(2026, 3, 1, 9, 30), id="a datetime"),
+ param('a "quoted" \\ mess', 'a "quoted" \\ mess', id="text needing json escaping"),
+ ],
+)
+def test_a_whole_string_placeholder_takes_the_values_own_type(loader, write, given, expected):
+ """The rule that lets a template stay valid JSON and still carry a non-string."""
+ write("order.json", '{"field": "{field}"}')
+
+ document = loader.load_document("order.json", field=given)
+
+ assert document["field"] == expected
+ assert type(document["field"]) is type(expected)
+
+
+def test_a_placeholder_inside_a_longer_string_is_interpolated_as_text(loader, write):
+ write("order.json", '{"reference": "REF-{n}/{suffix}"}')
+
+ assert loader.load_document("order.json", n=7, suffix="a")["reference"] == "REF-7/a"
+
+
+def test_text_interpolated_into_a_string_is_escaped_for_one(loader, write):
+ write("order.json", '{"reference": "REF-{n}"}')
+
+ assert loader.load_document("order.json", n='a "quoted" \\ mess')["reference"] == 'REF-a "quoted" \\ mess'
+
+
+def test_an_unquoted_placeholder_is_filled_too(loader, write):
+ """Not valid JSON before it is filled, which is why the quoted form is documented."""
+ write("order.json", '{"quantity": {quantity}}')
+
+ assert loader.load_document("order.json", quantity="3")["quantity"] == 3
+
+
+def test_one_placeholder_can_appear_more_than_once(loader, write):
+ write("order.json", '{"unique_id": "{name}", "slug": "order-{name}", "nested": {"also": "{name}"}}')
+
+ document = loader.load_document("order.json", name="a1")
+
+ assert document == {"unique_id": "a1", "slug": "order-a1", "nested": {"also": "a1"}}
+
+
+def test_a_placeholder_can_fill_an_extended_json_wrapper(loader, write):
+ """Filling the text inside ``$date``, for a value that arrives as a string."""
+ write("order.json", '{"placedAt": {"$date": "{placed_at}"}}')
+
+ document = loader.load_document("order.json", placed_at="2026-03-01T09:30:00Z")
+
+ assert document["placedAt"] == datetime.datetime(2026, 3, 1, 9, 30)
+
+
+def test_a_placeholder_can_stand_in_for_a_field_name(loader, write):
+ write("order.json", '{"{field}": "new"}')
+
+ assert loader.load_document("order.json", field="status") == {"status": "new"}
+
+
+def test_double_braces_inside_a_string_are_a_literal_brace(loader, write):
+ """How a document that genuinely holds ``{word}`` in a string says so."""
+ write("order.json", '{"template": "{{unfilled}}", "filled": "{filled}"}')
+
+ document = loader.load_document("order.json", filled="yes")
+
+ assert document == {"template": "{unfilled}", "filled": "yes"}
+
+
+def test_the_braces_json_itself_uses_are_left_alone(loader, write):
+ """Regression: unescaping ``}}`` outside a string would close nothing correctly.
+
+ A nested object ends in ``}}`` and an empty one is ``{}``. Neither is a placeholder,
+ and neither may be rewritten by the escape handling.
+ """
+ write("order.json", '{"a": {"b": {"c": 1}}, "empty": {}, "list": [{"sku": "A-1"}]}')
+
+ document = loader.load_document("order.json")
+
+ assert document == {"a": {"b": {"c": 1}}, "empty": {}, "list": [{"sku": "A-1"}]}
+
+
+def test_a_file_with_no_placeholders_and_no_arguments_is_unchanged(loader, write):
+ write("order.json", '{"status": "new", "lines": [{"sku": "A-1"}]}')
+
+ assert loader.load_document("order.json") == {"status": "new", "lines": [{"sku": "A-1"}]}
+
+
+def test_an_unfilled_placeholder_fails_with_the_file_and_the_holes(loader, write):
+ """The literal text ``{unique_id}`` is insertable, so leaving it would seed bad data."""
+ write("order.json", '{"unique_id": "{unique_id}", "customerId": "{customerId}", "status": "new"}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json", customerId=OID)
+
+ assert "order.json" in str(error.value)
+ assert "{unique_id}" in str(error.value)
+ assert "{customerId}" not in str(error.value)
+ assert "Given: customerId" in str(error.value)
+
+
+def test_every_unfilled_placeholder_is_reported_at_once(loader, write):
+ write("order.json", '{"a": "{first}", "b": "{second}"}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json")
+
+ assert "{first}, {second}" in str(error.value)
+ assert "Given: nothing" in str(error.value)
+
+
+def test_a_repeated_unfilled_placeholder_is_reported_once(loader, write):
+ write("order.json", '{"a": "{name}", "b": "order-{name}"}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json")
+
+ assert str(error.value).count("{name}") == 1
+
+
+def test_an_escape_in_the_files_own_json_survives_filling(loader, write):
+ r"""The scan has to step over ``\"`` rather than read it as the end of the string."""
+ write("order.json", r'{"note": "a \" quote, a \\ backslash and {n}"}')
+
+ assert loader.load_document("order.json", n=7)["note"] == 'a " quote, a \\ backslash and 7'
+
+
+def test_a_placeholder_wins_over_a_field_of_the_same_name(loader, write):
+ """The file declared the hole, so filling it is what was meant."""
+ write("order.json", '{"status": "{status}", "note": "unfilled is impossible"}')
+
+ assert loader.load_document("order.json", status="shipped")["status"] == "shipped"
+
+
+def test_an_argument_that_is_neither_a_placeholder_nor_a_field_names_both(loader, write):
+ write("order.json", '{"status": "{status}", "total": 1.0}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json", status="new", totl="2.0")
+
+ assert "order.json" in str(error.value)
+ assert "'totl' is neither" in str(error.value)
+ assert "Placeholders: {status}" in str(error.value)
+ assert "Fields: status, total" in str(error.value)
+
+
+def test_placeholders_and_dotted_overrides_work_in_one_call(loader, write):
+ write("order.json", '{"unique_id": "{unique_id}", "status": "new", "lines": [{"quantity": 1}]}')
+
+ document = loader.load_document("order.json", unique_id="a1", status="shipped",
+ **{"lines.0.quantity": "3"})
+
+ assert document == {"unique_id": "a1", "status": "shipped", "lines": [{"quantity": 3}]}
+
+
+def test_suite_variables_are_substituted_before_placeholders_are_filled(loader, write, suite_variables):
+ """So an argument's value is inserted last and is never resolved as a variable itself."""
+ suite_variables["${EMAIL}"] = "a@example.test"
+ write("order.json", '{"email": "${EMAIL}", "note": "{note}"}')
+
+ document = loader.load_document("order.json", note="${NOT_A_VARIABLE}")
+
+ assert document == {"email": "a@example.test", "note": "${NOT_A_VARIABLE}"}
+
+
+def test_braces_from_a_variables_value_are_data(loader, write, suite_variables):
+ """A placeholder is something the file says, so a value that holds braces holds text.
+
+ Substitution runs first, so without this the value would be scanned as a template and
+ the keyword would fail on a hole the file never declared.
+ """
+ suite_variables["${GREETING}"] = "Hi {first_name}"
+ write("order.json", '{"note": "${GREETING}"}')
+
+ assert loader.load_document("order.json") == {"note": "Hi {first_name}"}
+
+
+def test_braces_from_a_variables_value_are_not_filled_by_an_argument(loader, write, suite_variables):
+ """Even when an argument happens to be named after them, so the order cannot be used.
+
+ The name is no argument of the file's, so it is read as an override path instead,
+ which is what names the mistake.
+ """
+ suite_variables["${GREETING}"] = "Hi {first_name}"
+ write("order.json", '{"note": "${GREETING}"}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json", first_name="Ada")
+
+ assert "'first_name' is neither" in str(error.value)
+
+
+def test_a_placeholder_the_file_declares_is_still_filled_after_substitution(loader, write, suite_variables):
+ """The regression the two tests above must not cause."""
+ suite_variables["${EMAIL}"] = "a@example.test"
+ write("order.json", '{"email": "${EMAIL}", "status": "{status}"}')
+
+ document = loader.load_document("order.json", status="new")
+
+ assert document == {"email": "a@example.test", "status": "new"}
+
+
+def test_a_whole_file_variable_takes_its_arguments_as_overrides(loader, write, suite_variables):
+ """There is no text to fill, so every argument is a path into the object."""
+ suite_variables["${ORDER}"] = {"status": "new"}
+ write("order.json", "${ORDER}")
+
+ assert loader.load_document("order.json", status="shipped") == {"status": "shipped"}
+
+
+# --------------------------------------------------------------------------- #
+# Overrides
+# --------------------------------------------------------------------------- #
+
+def test_a_top_level_field_is_overridden(loader, write):
+ write("order.json", '{"status": "new", "total": 1.0}')
+
+ assert loader.load_document("order.json", status="shipped") == {"status": "shipped", "total": 1.0}
+
+
+def test_a_nested_field_is_overridden_without_rebuilding_the_document(loader, write):
+ write("order.json", '{"customer": {"name": "A", "address": {"city": "Utrecht"}}}')
+
+ document = loader.load_document("order.json", **{"customer.address.city": "Amsterdam"})
+
+ assert document == {"customer": {"name": "A", "address": {"city": "Amsterdam"}}}
+
+
+def test_a_list_position_is_written_as_a_number(loader, write):
+ write("recipe.json", '{"components": [{"evaporationFactor": 1.0}, {"evaporationFactor": 1.0}]}')
+
+ document = loader.load_document("recipe.json", **{"components.0.evaporationFactor": "0.8"})
+
+ assert document["components"][0]["evaporationFactor"] == 0.8
+ assert document["components"][1]["evaporationFactor"] == 1.0
+
+
+def test_a_whole_list_item_can_be_replaced(loader, write):
+ write("recipe.json", '{"components": [{"sku": "A"}, {"sku": "B"}]}')
+
+ document = loader.load_document("recipe.json", **{"components.1": '{"sku": "C"}'})
+
+ assert document["components"] == [{"sku": "A"}, {"sku": "C"}]
+
+
+@pytest.mark.parametrize(
+ "given, expected",
+ [
+ param("0.8", 0.8, id="a number is a number"),
+ param("3", 3, id="a whole number is an int"),
+ param("true", True, id="a boolean"),
+ param("null", None, id="null"),
+ param("shipped", "shipped", id="a word stays text"),
+ param("2026-03-01", "2026-03-01", id="a date-looking string stays text"),
+ param('{"$oid": "%s"}' % OID, ObjectId(OID), id="extended json"),
+ param('{"nested": 1}', {"nested": 1}, id="an object"),
+ ],
+)
+def test_an_override_written_as_text_is_read_like_the_files_own_values(loader, write, given, expected):
+ write("order.json", '{"field": "original"}')
+
+ assert loader.load_document("order.json", field=given)["field"] == expected
+
+
+def test_an_override_given_as_an_object_is_used_unchanged(loader, write):
+ """Robot Framework hands ``${oid}`` over as the object it is, not as text."""
+ write("order.json", '{"_id": "original"}')
+ object_id = ObjectId(OID)
+
+ assert loader.load_document("order.json", _id=object_id)["_id"] is object_id
+
+
+def test_an_override_can_target_an_update_operator(loader, write):
+ write("ship.json", '{"$set": {"status": "new"}}')
+
+ document = loader.load_document("ship.json", **{"$set.status": "shipped"})
+
+ assert document == {"$set": {"status": "shipped"}}
+
+
+@pytest.mark.parametrize(
+ "path, expected_in_message",
+ [
+ param("customer.name", "Available: status, lines", id="a field that is not there at all"),
+ param("lines.0.qty", "Available: sku", id="a misspelled field inside a list item"),
+ param("lines.5.sku", "holds 1 items", id="a list index out of range"),
+ param("lines.first.sku", "has to be a number", id="a list step that is not a number"),
+ param("status.upper", "is a str", id="a step into a plain value"),
+ param("lines.5", "holds 1 items", id="a list index out of range at the end"),
+ ],
+)
+def test_an_override_path_that_does_not_exist_fails_with_what_was_there(loader, write, path, expected_in_message):
+ write("order.json", '{"status": "new", "lines": [{"sku": "A-1"}]}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json", **{path: "x"})
+
+ assert "order.json" in str(error.value)
+ assert path in str(error.value)
+ assert expected_in_message in str(error.value)
+
+
+def test_an_override_path_through_a_null_field_is_reported(loader, write):
+ write("order.json", '{"customer": null}')
+
+ with pytest.raises(ValueError, match="is null"):
+ loader.load_document("order.json", **{"customer.name": "A"})
+
+
+@pytest.mark.parametrize(
+ "path, expected_in_message",
+ [
+ param("address.city.postcode", "'address' is null", id="through a null field"),
+ param("name.first.initial", "'name' is a str", id="through a plain value"),
+ ],
+)
+def test_a_path_that_keeps_going_past_a_value_it_cannot_enter_is_reported(loader, write, path,
+ expected_in_message):
+ write("order.json", '{"name": "A", "address": null}')
+
+ with pytest.raises(ValueError) as error:
+ loader.load_document("order.json", **{path: "x"})
+
+ assert expected_in_message in str(error.value)
+
+
+def test_a_null_field_can_itself_be_overridden(loader, write):
+ write("order.json", '{"note": null}')
+
+ assert loader.load_document("order.json", note="late")["note"] == "late"
+
+
+def test_the_file_on_disk_is_not_changed_by_overrides(loader, write, documents):
+ name = write("order.json", '{"status": "new"}')
+ loader.load_document(name, status="shipped")
+
+ assert loader.load_document(name) == {"status": "new"}
+ assert (documents / name).read_text(encoding="utf-8") == '{"status": "new"}'
+
+
+# --------------------------------------------------------------------------- #
+# What the keyword hands back
+# --------------------------------------------------------------------------- #
+
+def test_the_document_is_returned_for_dotted_access_in_a_suite(loader, write):
+ write("order.json", '{"customer": {"name": "A"}, "lines": [{"sku": "A-1"}]}')
+
+ document = loader.load_document("order.json")
+
+ assert isinstance(document, DotDict)
+ assert document.customer.name == "A"
+ assert document.lines[0].sku == "A-1"
+
+
+# --------------------------------------------------------------------------- #
+# Insert Document From File
+# --------------------------------------------------------------------------- #
+
+@pytest.fixture
+def mongo_loader(mongo, documents):
+ """Keywords backed by an in-memory MongoDB that also read from the document directory."""
+ mongo.document_path = documents
+ return mongo
+
+
+def test_a_document_is_read_and_inserted_in_one_step(mongo_loader, write):
+ write("order.json", '{"_id": {"$oid": "%s"}, "status": "new"}' % OID)
+
+ doc_id = mongo_loader.insert_document_from_file("orders", "order.json")
+
+ assert doc_id == ObjectId(OID)
+ assert mongo_loader.find_document("orders", _id=OID)["status"] == "new"
+
+
+def test_inserting_from_a_file_applies_the_overrides(mongo_loader, write):
+ write("order.json", '{"status": "new", "lines": [{"quantity": 1}]}')
+
+ mongo_loader.insert_document_from_file("orders", "order.json", status="shipped",
+ **{"lines.0.quantity": "3"})
+
+ stored = mongo_loader.find_document("orders", status="shipped")
+ assert stored["lines"][0]["quantity"] == 3
+
+
+def test_inserting_from_a_file_fills_its_placeholders(mongo_loader, write):
+ """The dominant shape: one template, a different document inserted per call."""
+ write("order.json", '{"unique_id": "{unique_id}", "customerId": "{customerId}", "status": "new"}')
+
+ for unique_id in ("order-1", "order-2"):
+ mongo_loader.insert_document_from_file("orders", "order.json", unique_id=unique_id,
+ customerId=ObjectId(OID))
+
+ # Queried as an ObjectId, since only ``_id`` is coerced from a string. That it matches
+ # at all is the point: the placeholder stored a real ObjectId, not its text.
+ assert mongo_loader.count_documents("orders", customerId=ObjectId(OID)) == 2
+ assert mongo_loader.find_document("orders", unique_id="order-2")["status"] == "new"
+
+
+def test_inserting_from_a_file_fails_on_an_unfilled_placeholder(mongo_loader, write):
+ write("order.json", '{"unique_id": "{unique_id}"}')
+
+ with pytest.raises(ValueError, match="{unique_id}"):
+ mongo_loader.insert_document_from_file("orders", "order.json")
+
+ assert mongo_loader.count_documents("orders") == 0
+
+
+def test_inserting_from_a_file_uses_the_given_alias(mongo_loader, write, database):
+ write("order.json", '{"status": "new"}')
+ mongo_loader.connection_manager.add_to_connection_pool(database, "other")
+
+ mongo_loader.insert_document_from_file("orders", "order.json", alias="other")
+
+ assert mongo_loader.count_documents("orders", alias="other", status="new") == 1
+ assert mongo_loader.count_documents("orders", status="new") == 0
+
+
+def test_a_missing_file_fails_before_anything_is_inserted(mongo_loader):
+ with pytest.raises(ValueError, match="missing.json"):
+ mongo_loader.insert_document_from_file("orders", "missing.json")
+
+ assert mongo_loader.count_documents("orders") == 0