Per-user data directories, one process-safe log, no import side effects

CLAUDE.md

@@ -46,9 +46,10 @@ installed, so all drivers whose dependencies are on PyPI get import-tested.
 - **`Pyro5.config` is reset around every test** (autouse `pyro_config` fixture), because
   `update_pyro_config()` writes Pyro5 process globals and importing `pyrolab.api` applies the developer's
   own user config. Still pass `host="localhost"` explicitly when constructing daemons.
-- **Use the `data_dir` fixture for anything touching runtime files** (`USER_CONFIG_FILE`, `RUNTIME_CONFIG`,
-  `LOCKFILE`, `PYROLAB_LOGDIR`, `NAMESERVER_STORAGE`). Tests must never write to the real data dir. See
-  "Runtime state" for why patching `pyrolab.X` alone is not enough.
+- **The `data_dir` fixture is autouse**: every test sees PyroLab's paths (`USER_CONFIG_FILE`, `RUNTIME_CONFIG`,
+  `LOCKFILE`, `PYROLAB_LOGFILE`, `NAMESERVER_STORAGE`, `LEGACY_DATA_DIR`, ...) redirected into its own temp dir, so
+  no test can touch the real per-user directories or copy a developer's real pre-0.5 files. Add new path
+  constants to it. See "Runtime state" for why patching `pyrolab.X` alone is not enough.
 - `global_config` gives a fresh `GlobalConfiguration` singleton; `sample_config_yaml` / `sample_config_file`
   are a complete valid config using only `pyrolab.server` and `pyrolab.drivers.sample`.
 - **Hardware tests** get `@pytest.mark.hardware` and are skipped unless `--run-hardware` is passed. Import
@@ -145,21 +146,34 @@ Most commands are thin wrappers that proxy to the running `PyroLabDaemon`.
 
 ### Runtime state
 
-`pyrolab/__init__.py` creates and exports the data-dir constants at import time: `PYROLAB_DATA_DIR`
-(`pyrolab/data/local/` inside the package), `USER_CONFIG_FILE`, `RUNTIME_CONFIG`, `LOCKFILE`,
-`NAMESERVER_STORAGE`, `PYROLAB_LOGDIR`. It also installs a rotating file handler writing to
-`logs/pyrolab_<pid>.log`. Env vars: `PYROLAB_LOGLEVEL`, `PYROLAB_LOGFILE`, `PYROLAB_HUSH_DEPRECATION`,
-`PYROLAB_NO_VERSION_CHECK`.
+`pyrolab/__init__.py` exports the path constants, computed at import from `pyrolab/locations.py` (per-user
+`platformdirs` directories, or everything under `PYROLAB_DATA_DIR`): `USER_CONFIG_FILE`, `RUNTIME_CONFIG`,
+`LOCKFILE`, `STARTUP_ERROR_FILE`, `NAMESERVER_STORAGE`, `PYROLAB_LOGDIR`, `PYROLAB_LOGFILE`, `UPDATE_CHECK_FILE`.
+**Importing creates nothing** (`test_locations.py` checks this in a fresh interpreter); directories are created
+where files are written (`atomic_write_text` makes parents). Before 0.5 everything lived in the package's
+`pyrolab/data/local`; `migrate_legacy_files()` copies the config and nameserver databases from there once (a
+marker file stops it repeating), run by the CLI callback and the daemon.
+
+Other modules bind these constants by value (`from pyrolab import LOCKFILE, ...` in `api`, `cli`,
+`configure`, `manager`, `pyrolabd`), so redirecting a path means patching it in each of those modules —
+which is what the `data_dir` test fixture does.
+
+**Logging** (`pyrolab/logs.py`): as a library, `pyrolab` only adds a `NullHandler` — it never configures
+logging, `sys.excepthook`, or warning filters on import. The daemon (`pyrolabd.main`) is the **only writer** of
+one rotating JSON-lines log (`PYROLAB_LOGFILE`, 5 MB x 5). Each child runner logs through its **own pipe**
+(`log_to_pipe` in the child, `collect_from_pipe` in the parent): never a shared `multiprocessing.Queue` or
+loguru `enqueue=True`, because a child killed mid-write would hold the shared lock and block all logging (a
+real deadlock, tested in `test_logs.py`). The CLI's `run()` entry point (not `app`, which tests invoke)
+applies process-wide settings; CLI commands only log to stderr when `PYROLAB_LOGLEVEL` is set. `pyrolab
+status` / `logs show|export|clean` read the log back; entries after the last `daemon-start` event belong to
+the current run. Env vars: `PYROLAB_DATA_DIR`, `PYROLAB_LOGFILE`, `PYROLAB_LOGLEVEL`,
+`PYROLAB_HUSH_DEPRECATION`, `PYROLAB_NO_VERSION_CHECK`.
 
 **Importing `pyrolab` must never touch the network.** The PyPI update check lives in `pyrolab/updates.py`
 and runs only from the CLI's main callback, at most once a day (result cached in `UPDATE_CHECK_FILE`).
 An autouse test fixture sets `PYROLAB_NO_VERSION_CHECK` so CLI tests stay offline, and
 `test_updates.py` fails if importing the package performs any hostname lookup.
 
-Other modules bind these constants by value (`from pyrolab import LOCKFILE, ...` in `api`, `cli`,
-`configure`, `manager`, `pyrolabd`), so redirecting a path means patching it in each of those modules —
-which is what the `data_dir` test fixture does.
-
 `pyrolab/api.py` is the intended public surface — it re-exports the Pyro5 names plus PyroLab's, and on import
 applies the first configured nameserver's settings to `Pyro5.config` so `locate_ns()` just works. That step is
 best-effort: **importing `pyrolab.api` must never fail** because of the local config (invalid, no nameservers,
@@ -175,8 +189,11 @@ excluded from import sorting (a ruff per-file ignore); import order there is loa
   `connect()` arguments must be keyword arguments with defaults — `autoconnect()` dictionary-unpacks
   `_autoconnect_params` into it.
 - numpydoc docstrings and type hints throughout; the API docs are autogenerated from them.
-- Log via the `logging` module (`log = logging.getLogger(__name__)`). Servers run windowless in child
-  processes and fail silently, so logs are often the only diagnostic.
+- Log via the standard `logging` module (`log = logging.getLogger(__name__)`), never by configuring handlers
+  in library code. Servers run windowless in child processes and fail silently, so logs (`pyrolab status`,
+  `pyrolab logs show`) are often the only diagnostic.
+- `host: public` means "the address clients should reach this machine at" (`utils.get_ip`, which falls back
+  rather than failing without a route outside). Clients connect to the PyroLab server, never to instruments.
 - Optional hardware dependencies belong in a pyproject extra (`tsl550`, `ppcl55x`, `rto`, `arduino`,
   `cameras`), not in the base `dependencies`. PyPI rejects direct git URLs, so git-sourced extras stay
   commented out. `[tool.uv.sources]` can point development installs at git, but PyPI users never see it, so

docs/user_guide/clients.rst

@@ -16,6 +16,17 @@ up the connection:
     with Proxy(uri) as remote_object:
         remote_object.useful_method()
 
+To see where an error happened on the server, not just that it was raised,
+have Python print remote tracebacks (PyroLab doesn't change this for you when
+imported, since it's your application's setting):
+
+.. code-block:: python
+
+    import sys
+    import Pyro5.errors
+
+    sys.excepthook = Pyro5.errors.excepthook
+
 Assuming drivers and services have been written correctly, they only raise
 builtin Python exceptions. You can also catch different levels of exceptions if
 you know what exceptions an object can raise. Therefore you can use them in

docs/user_guide/configuration.rst

@@ -51,3 +51,24 @@ Services
 .. todo::
 
     Coming soon.
+
+
+Choosing ``host``
+-----------------
+
+A nameserver or daemon listens on the address given by ``host``, and the URIs
+it publishes contain that address, so it must be one that *clients* can reach.
+Clients only ever connect to the PyroLab machine; instruments are reached by
+that machine, not by clients.
+
+``localhost``
+    Only programs on the same machine can connect (the default).
+``public``
+    PyroLab picks this machine's address on the network that routes to the
+    internet. On a machine without a route outside (an isolated lab network)
+    it falls back to the address the machine's hostname resolves to, then to
+    ``127.0.0.1``, and logs a warning.
+an IP address or hostname
+    Use this when the machine is on several networks and the clients are not
+    on the internet-facing one, e.g. a private lab network on a second
+    network card.

docs/user_guide/envvars.rst

@@ -1,31 +1,92 @@
 .. _user_guide:
 
 
-Environment Variables
-=====================
-
-A few notes on simply starting PyroLab. When launched from the terminal,
-PyroLab reads a few environment variables for convenient configuration.
+Files, Logs, and Environment Variables
+======================================
 
-.. todo::
-
-   Link to where the data directories is discussed, for appdirs.
+Where PyroLab keeps its files
+-----------------------------
 
+Each user has their own PyroLab configuration, logs, and runtime state, kept in
+the platform's usual places rather than inside the installed package (so
+upgrading PyroLab never touches them):
+
+.. list-table::
+   :header-rows: 1
+
+   * -
+     - Linux
+     - Windows
+     - macOS
+   * - Configuration
+     - ``~/.config/pyrolab``
+     - ``%LOCALAPPDATA%\pyrolab``
+     - ``~/Library/Application Support/pyrolab``
+   * - Logs
+     - ``~/.local/state/pyrolab/log``
+     - ``%LOCALAPPDATA%\pyrolab\Logs``
+     - ``~/Library/Logs/pyrolab``
+   * - Runtime state
+     - ``~/.local/state/pyrolab``
+     - ``%LOCALAPPDATA%\pyrolab``
+     - ``~/Library/Application Support/pyrolab``
+   * - Nameserver data
+     - ``~/.local/share/pyrolab``
+     - ``%LOCALAPPDATA%\pyrolab``
+     - ``~/Library/Application Support/pyrolab``
+
+``pyrolab --data`` prints the exact paths on your machine. Set
+``PYROLAB_DATA_DIR`` (below) to keep everything in one directory instead.
+
+Versions before 0.5 kept these files inside the installed package. The first
+time you run a ``pyrolab`` command after upgrading, it copies your
+configuration and nameserver databases to the new locations (once, without
+overwriting anything, and leaving the originals in place).
+
+
+Logs
+----
+
+The background daemon writes a single log file, ``pyrolab.log``, for itself and
+every nameserver and daemon it runs. It rotates at 5 MB, keeping five older
+files. Each line is one JSON record. Rather than reading it directly:
+
+``pyrolab status``
+   Whether the daemon is running, and the warnings and errors logged since it
+   started.
+``pyrolab logs show``
+   The log since the daemon last started (``--level``, ``--all``, ``--lines``).
+``pyrolab logs export FILE``
+   The whole log as text (or ``--json``), to attach to a bug report.
+``pyrolab logs clean``
+   Delete the logs (stop the daemon first).
+
+Output the daemon itself prints (such as a crash in a child process before it
+could log) goes to ``pyrolabd_output.log`` beside the log.
+
+When you use PyroLab as a library (in a client script), it writes no log
+files and leaves your application's logging alone; its messages go wherever
+your application sends the ``pyrolab`` loggers.
+
+
+Environment variables
+---------------------
+
+``PYROLAB_DATA_DIR``
+   Keep all of PyroLab's files in this directory (logs in its ``logs``
+   subdirectory) instead of the per-user locations above.
 ``PYROLAB_LOGFILE``
-   The log file to write to. If not set, the default is ``pyrolab.log``
-   within PyroLab's default data directory (see appdirs).
+   Where the daemon writes its log, instead of ``pyrolab.log`` in the log
+   directory.
 ``PYROLAB_LOGLEVEL``
-   The log level to use. If not set, the default is ``INFO``.
+   The level to log at (``DEBUG``, ``INFO``, ``WARNING``, ...; default
+   ``INFO``). For the daemon this sets what goes into the log; when set, other
+   ``pyrolab`` commands also print their own log messages to the terminal.
 ``PYROLAB_HUSH_DEPRECATION``
-   If set, silences all DeprecationWarnings from all modules.
+   If set, the ``pyrolab`` command and daemon hide DeprecationWarnings.
 ``PYROLAB_NO_VERSION_CHECK``
    If set (to any value), the ``pyrolab`` command line tool never checks PyPI
    for a newer release. Otherwise it checks at most once a day, with a short
    timeout, and caches the result (including a failed check) so that machines
    on isolated networks don't wait on every command. Importing ``pyrolab`` as
    a library never checks.
-
-This can be very helpful when debugging to see how the program is progressing,
-and when the program is being too verbose, it's easy to silence. Note that
-including debugging will also include log statements from all of PyroLab's
-dependencies.

pyproject.toml

@@ -70,6 +70,7 @@ dependencies = [
   "tabulate",
   "requests",
   "packaging>=23.0",
+  "platformdirs>=3.0",
 ]
 
 # Optional hardware dependencies, installed with the "extras" syntax, e.g.
@@ -113,7 +114,7 @@ namespaces = false
 "Research Group" = "https://camacholab.byu.edu/"
 
 [project.scripts]
-pyrolab = "pyrolab.cli:app"
+pyrolab = "pyrolab.cli:run"
 
 [tool.pytest.ini_options]
 minversion = "6.0"

pyrolab/__init__.py

@@ -23,16 +23,10 @@
 A framework for using remote lab instruments as local resources built on Pyro5.
 """
 
-import atexit
-import fileinput
+import logging
 import os
-import pathlib
 import platform
-import re
 import sys
-from pathlib import Path
-from time import strptime
-from typing import Callable, Iterable
 
 # Check if Python version is supported
 pyversion = sys.version_info
@@ -64,105 +58,33 @@
 __website_url__ = "https://camacholab.byu.edu/"
 
 
-# Filter warnings
-import warnings
-
-warnings.filterwarnings("default", category=DeprecationWarning)
-if "PYROLAB_HUSH_DEPRECATION" in os.environ:
-    warnings.filterwarnings("ignore", category=DeprecationWarning)
-
-
-# Configuration directories
-from importlib.resources import files
-
-base_path = files("pyrolab") / "data" / "local"
-
-# Data directories
-PYROLAB_DATA_DIR = pathlib.Path(base_path)
-PYROLAB_DATA_DIR.mkdir(parents=True, exist_ok=True)
-
-NAMESERVER_STORAGE = PYROLAB_DATA_DIR / "nameserver"
-NAMESERVER_STORAGE.mkdir(parents=True, exist_ok=True)
-
-PYROLAB_LOGDIR = PYROLAB_DATA_DIR / "logs"
-PYROLAB_LOGDIR.mkdir(parents=True, exist_ok=True)
-PYROLAB_MASTERLOG = PYROLAB_LOGDIR / "pyrolab.log"
+# Where PyroLab keeps its files: per-user directories (see pyrolab.locations).
+# Importing creates nothing; files and directories are created when written.
+from pathlib import Path
 
-LOCKFILE = PYROLAB_DATA_DIR / "pyrolabd.lock"
-USER_CONFIG_FILE = PYROLAB_DATA_DIR / "user_configuration.yaml"
-RUNTIME_CONFIG = PYROLAB_DATA_DIR / "runtime_config.yaml"
-UPDATE_CHECK_FILE = PYROLAB_DATA_DIR / "update_check.json"
+from pyrolab.locations import LEGACY_DATA_DIR
+from pyrolab.locations import locations as _find_locations
+
+_locations = _find_locations()
+
+PYROLAB_DATA_DIR = _locations.data_dir
+NAMESERVER_STORAGE = _locations.data_dir / "nameserver"
+PYROLAB_LOGDIR = _locations.log_dir
+# The daemon's single log file; PYROLAB_LOGFILE overrides where it goes.
+PYROLAB_LOGFILE = Path(
+    os.environ.get("PYROLAB_LOGFILE") or PYROLAB_LOGDIR / "pyrolab.log"
+)
+USER_CONFIG_FILE = _locations.config_dir / "user_configuration.yaml"
+LOCKFILE = _locations.state_dir / "pyrolabd.lock"
+RUNTIME_CONFIG = _locations.state_dir / "runtime_config.yaml"
 # Why the background daemon exited before it could serve requests, for `up`.
-STARTUP_ERROR_FILE = PYROLAB_DATA_DIR / "pyrolabd_startup_error.txt"
-
-
-# Set up logging to file
-import logging
-import logging.handlers
-
-
-def get_loglevel() -> int:
-    loglevel = os.getenv("PYROLAB_LOGLEVEL", "INFO")
-    try:
-        loglevel = getattr(logging, loglevel.upper())
-    except AttributeError:
-        loglevel = logging.INFO
-    return loglevel
-
-
-if len(logging.root.handlers) == 0:
-    logfile = PYROLAB_LOGDIR / f"pyrolab_{os.getpid()}.log"
-    root = logging.getLogger()
-    h = logging.handlers.RotatingFileHandler(logfile, "a", 30000, 10)
-    f = logging.Formatter(
-        "[%(asctime)s.%(msecs)03d] %(levelname)-8s %(process)-5s %(processName)-16s %(name)-20s %(message)s",
-        datefmt="%Y-%m-%d %H:%M:%S",
-    )
-    h.setFormatter(f)
-    root.addHandler(h)
-    root.setLevel(get_loglevel())
-    root.debug("PyroLab logging configured")
-
-
-# Include remote traceback in local tracebacks
-import Pyro5.errors
-
-sys.excepthook = Pyro5.errors.excepthook
-
-
-def try_itr(func: Callable, itr: Iterable, *exceptions, **kwargs):
-    """
-    Tests a function on an iterable, yields iterable if no exception is raised.
-    """
-    for elem in itr:
-        try:
-            func(elem, **kwargs)
-            yield elem
-        except exceptions:
-            pass
-
+STARTUP_ERROR_FILE = _locations.state_dir / "pyrolabd_startup_error.txt"
+# Records that files from a pre-0.5 install were copied (see pyrolab.locations).
+LEGACY_MIGRATION_MARKER = _locations.state_dir / "legacy_files_copied.txt"
+UPDATE_CHECK_FILE = _locations.cache_dir / "update_check.json"
 
-# @atexit.register
-# def condense_logs():
-#     """
-#     Cleans up the logfile directory every once in a while.
 
-#     Runs at exit or when "pyrolab down" is called. Condenses all process log
-#     files into a single logfile that is sorted by timestamp.
-#     """
-#     f_names = list(PYROLAB_LOGDIR.glob("*.*"))
-#     lines = list(fileinput.input(f_names))
-#     t_fmt = "%Y-%m-%d %H:%M:%S.%f"  # format of time stamps
-#     t_pat = re.compile(r"\[(.+?)\]")  # pattern to extract timestamp
-#     lines = list(
-#         try_itr(
-#             lambda l: strptime(t_pat.search(l).group(1), t_fmt), lines, AttributeError
-#         )
-#     )
-#     with PYROLAB_MASTERLOG.open(mode="w") as f:
-#         for l in sorted(lines, key=lambda l: strptime(t_pat.search(l).group(1), t_fmt)):
-#             f.write(l)
-#     if PYROLAB_MASTERLOG in f_names:
-#         f_names.remove(PYROLAB_MASTERLOG)
-#     for f in f_names:
-#         f.unlink()
+# As a library, PyroLab leaves logging to the application: its loggers do
+# nothing until the application (or PyroLab's own CLI and daemon, see
+# pyrolab.logs) configures logging.
+logging.getLogger(__name__).addHandler(logging.NullHandler())