Support a virtual environment

[SwooshyCueb]

Supporting a virtual environment would allow the python plugin to use a different set of modules than are installed in the system's site-packages. This would allow iRODS to make use of newer or different versions of modules that would otherwise be provided by the distro repositories, without causing problems for the distro's package manager. We should implement optional support for python virtual environments. This will require some new configuration options for the plugin.

Prerequisite: #211

Relevant documentation:

• https://docs.python.org/3/library/sys_path_init.html
• https://docs.python.org/3/c-api/init_config.html#init-path-config

[trel]

can we do this with a compiled boost.python? great if true.

[SwooshyCueb]

From what I can tell, yes. We just have to tweak how we hold it

[trel]

then that is extremely valuable / powerful.

[SwooshyCueb]

We have a few curveballs to contend with.

First, PyConfig isn't a thing prior to Python 3.8. EL8 is the only distro we support that has a version of Python old enough for this to matter. I haven't really started looking into a solution for this beyond simply disabling the interpreter configuration functionality if the Python version being built against is too old.

Second, there doesn't seem to be a way to manipulate the module search path list in the way we need without replacing it outright. This means that we need to populate the list with the default system module search paths before we add our own. For Python 3.8, 3.9, and 3.10, this is fairly straightforward, as we can call PyConfig_Read(), which will populate the PyConfig object with the default paths, minus any paths added by site. However, starting in 3.11, this functionality was removed from PyConfig_Read().

The documentation for PyConfig.module_search_paths_set states that Py_InitializeFromConfig() will populate module_search_paths, which may have been a solution to explore for 3.11+, but as far as I can tell, this isn't actually true. In my testing, both module_search_paths and module_search_paths_set are the same before and after a call to Py_InitializeFromConfig(). I've opened an issue about that at python/cpython#122963.

One solution might be to initialize the interpreter with a mostly-default configuration, slurp sys.path, and then use that to re-initialize the interpreter. However, this seems icky.