Test collection to broad, walking sideways, causes PermissionError for tests in /tmp subdirs #12294

ibressler · 2024-05-07T14:39:25Z

a detailed description of the bug or problem you are having
output of pip list from the virtual environment you are using
pytest and operating system versions
minimal example if possible

Hello, I am a user of pytest and pytest-notebook. I am not sure if this qualifies as a bug here or for pytest-notebook. But since it seems to be very much related to the plugin system, collection and path handling, I post this here. Plus, it seems very similar or related to #11781.

When running a test with a pytester fixture (formerly testdir) a temporary directory is created and used under /tmp. Here, that would be something like /tmp/pytest-of-ingo/pytest-44/test_run_fail0. In the collection phase pytest looks for tests in just /tmp and stumbles on systemd related directories which are inaccessible, causing a PermissionError: [Errno 13] Permission denied: to stop it.

PermissionError details/traceback

-------------------------------------------------------------- Captured stdout call --------------------------------------------------------------
TEST /tmp/pytest-of-ingo/pytest-44/test_run_fail0
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-8.3.0.dev28+g4080459f0, pluggy-1.5.0 -- /home/ingo/.py11venv/bin/python3
cachedir: .pytest_cache
NB exec dir: /home/ingo/code/3rdp/pytest-notebook/tests/raw_files
rootdir: /home/ingo/code/3rdp/pytest-notebook
configfile: tox.ini
plugins: cov-4.1.0, anyio-4.3.0, dash-2.15.0, pytest_notebook-0.10.0
collecting ... collected 0 items / 1 error

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
/home/ingo/.py11venv/lib/python3.11/site-packages/pluggy/_hooks.py:513: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
/home/ingo/.py11venv/lib/python3.11/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/home/ingo/.py11venv/lib/python3.11/site-packages/pluggy/_manager.py:480: in traced_hookexec
    return outcome.get_result()
/home/ingo/.py11venv/lib/python3.11/site-packages/pluggy/_manager.py:477: in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
/home/ingo/.py11venv/lib/python3.11/site-packages/_pytest/python.py:179: in pytest_collect_directory
    if pkginit.is_file():
/usr/lib/python3.11/pathlib.py:1268: in is_file
    return S_ISREG(self.stat().st_mode)
/usr/lib/python3.11/pathlib.py:1014: in stat
    return os.stat(self, follow_symlinks=follow_symlinks)
E   PermissionError: [Errno 13] Permission denied: '/tmp/systemd-private-36e9322d52084bf7bb27cacad8f3ac9e-ModemManager.service-S0fIze/__init__.py'
=========================== short test summary info ============================
ERROR ../../../../home/ingo/code/3rdp/pytest-notebook/::tmp - PermissionError...
=============================== 1 error in 0.12s ===============================
-------------------------------------------------------------- Captured stderr call --------------------------------------------------------------
ERROR: found no collectors for /tmp/pytest-of-ingo/pytest-44/test_run_fail0

But this seems to be the case only for those tests which are run in a different working directory, because the failing tests get the pytest-notebook specific flag --nb-exec-cwd which is forwarded to nbclient and sets the path where the notebook is run.
But this seems to affect the setting of PluginManager._confcutdir which in turn is used to restrict the paths for test collection. The pytest_load_initial_conftests hook seems to be run for the failing tests only and sets the _confcutdir to the cwd folder which might be elsewhere (/home for example) than the tmpdir used by the fixture. In pluginmanager._set_initial_conftests is gets:
invocation_dir=PosixPath('/tmp/pytest-of-ingo/pytest-44/test_run_fail0')
and confcutdir='/home/ingo/code/3rdp/pytest-notebook'
But I can't find out why this happens?

This took me some hours to trace back due to multiple levels of hooks being run (and am I far from familiar with how pytest works internally). And perhaps, someone with more experience with the internals of pytest can tell why _confcutdir is set or not set accordingly.

The test I refer here, is a rather simple example already. It belongs to the pytest-notebook extension. Cloning that and running pytest -v -k test_run_fail should trigger that error while pytest -v -k test_setup_with_skip_meta from the same file works.

I am on Ubuntu 23.10 locally with Python 3.11.6 and the following pip packages in my venv:

Package                   Version
------------------------- ----------------------
anyio                     4.3.0
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
attrs                     22.2.0
beautifulsoup4            4.12.3
bleach                    6.1.0
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        3.3.2
colorama                  0.4.6
coverage                  7.5.1
defusedxml                0.7.1
fastjsonschema            2.19.1
fqdn                      1.5.1
gitdb                     4.0.11
GitPython                 3.1.43
idna                      3.7
iniconfig                 2.0.0
isoduration               20.11.0
Jinja2                    3.1.4
jsonpointer               2.4
jsonschema                4.22.0
jsonschema-specifications 2023.12.1
jupyter_client            8.6.1
jupyter_core              5.7.2
jupyter-events            0.10.0
jupyter_server            2.14.0
jupyter-server-mathjax    0.2.6
jupyter_server_terminals  0.5.3
jupyterlab_pygments       0.3.0
MarkupSafe                2.1.5
mistune                   3.0.2
nbclient                  0.5.13
nbconvert                 7.16.4
nbdime                    4.0.1
nbformat                  5.10.4
nest-asyncio              1.6.0
overrides                 7.7.0
packaging                 24.0
pandocfilters             1.5.1
pip                       23.2
platformdirs              4.2.1
pluggy                    1.5.0
prometheus_client         0.20.0
ptyprocess                0.7.0
pycparser                 2.22
Pygments                  2.18.0
pytest                    8.3.0.dev28+g4080459f0
pytest_notebook           0.10.0
python-dateutil           2.9.0.post0
python-json-logger        2.0.7
PyYAML                    6.0.1
pyzmq                     26.0.3
referencing               0.35.1
requests                  2.31.0
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rpds-py                   0.18.1
Send2Trash                1.8.3
setuptools                68.1.2
six                       1.16.0
smmap                     5.0.1
sniffio                   1.3.1
soupsieve                 2.5
terminado                 0.18.1
tinycss2                  1.3.0
tornado                   6.4
traitlets                 5.14.3
types-python-dateutil     2.9.0.20240316
uri-template              1.3.0
urllib3                   2.2.1
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.8.0

But this happens with the Github Ubuntu runner as well: https://github.com/chrisjsewell/pytest-notebook/actions/runs/8976635149/job/24653858224#step:5:35

Here, it complains just about a different path being inaccessible (`/tmp/snap-private-tmp/__init__.py`).

Run pytest --durations=10 --cov=pytest_notebook --cov-report=xml --cov-report=term-missing --nb-coverage
============================= test session starts ==============================
platform linux -- Python 3.11.9, pytest-8.2.0, pluggy-1.5.0
rootdir: /home/runner/work/pytest-notebook/pytest-notebook
configfile: tox.ini
plugins: anyio-4.3.0, regressions-2.5.0, datadir-1.5.0, cov-5.0.0, pytest_notebook-0.10.0
collected 73 items

tests/test_cell_diff.py ................                                 [ 21%]
tests/test_coalesce_streams.py ..                                        [ 24%]
tests/test_execution.py ...                                              [ 28%]
tests/test_filter_diff.py .....                                          [ 35%]
tests/test_ipy_magic.py ...........                                      [ 50%]
tests/test_nb_diff.py ...                                                [ 54%]
tests/test_nb_regression.py .....                                        [ 61%]
tests/test_notebook.py ........                                          [ 72%]
tests/test_plugin_collector.py ..FF                                      [ 78%]
tests/test_plugin_fixture.py ..........                                  [ 91%]
tests/test_postprocessors/test_beautifulsoup.py ..                       [ 94%]
tests/test_postprocessors/test_blacken_code.py ..                        [ 97%]
tests/test_postprocessors/test_plugins.py .                              [ 98%]
tests/test_utils.py s                                                    [100%]

=================================== FAILURES ===================================
________________________________ test_run_fail _________________________________

testdir = <Testdir local('/tmp/pytest-of-runner/pytest-0/test_run_fail0')>

    def test_run_fail(testdir):
        copy_nb_to_tempdir("different_outputs_altered.ipynb")
        result = testdir.runpytest(
            "--nb-exec-cwd", os.path.join(PATH, "raw_files"), "--nb-test-files", "-v"
        )
        # fnmatch_lines does an assertion internally
>       result.stdout.fnmatch_lines(
            ["*::nbregression(test_nb) FAILED*", "*CellExecutionError:*"]
        )
E       Failed: nomatch: '*::nbregression(test_nb) FAILED*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.11.9, pytest-8.2.0, pluggy-1.5.0 -- /opt/hostedtoolcache/Python/3.11.9/x64/bin/python'
E           and: 'cachedir: .pytest_cache'
E           and: 'NB exec dir: /home/runner/work/pytest-notebook/pytest-notebook/tests/raw_files'
E           and: 'rootdir: /home/runner/work/pytest-notebook/pytest-notebook'
E           and: 'configfile: tox.ini'
E           and: 'plugins: anyio-4.3.0, regressions-2.5.0, datadir-1.5.0, cov-5.0.0, pytest_notebook-0.10.0'
E           and: 'collecting ... collected 0 items / 1 error'
E           and: ''
E           and: '==================================== ERRORS ===================================='
E           and: '________________________ ERROR collecting test session _________________________'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pluggy/_hooks.py:513: in __call__'
E           and: '    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pluggy/_manager.py:120: in _hookexec'
E           and: '    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pluggy/_manager.py:480: in traced_hookexec'
E           and: '    return outcome.get_result()'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pluggy/_manager.py:477: in <lambda>'
E           and: '    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/_pytest/python.py:179: in pytest_collect_directory'
E           and: '    if pkginit.is_file():'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/pathlib.py:1267: in is_file'
E           and: '    return S_ISREG(self.stat().st_mode)'
E           and: '/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/pathlib.py:1013: in stat'
E           and: '    return os.stat(self, follow_symlinks=follow_symlinks)'
E           and: "E   PermissionError: [Errno 13] Permission denied: '/tmp/snap-private-tmp/__init__.py'"
E           and: '=========================== short test summary info ============================'
E           and: "ERROR ../../../../home/runner/work/pytest-notebook/pytest-notebook/::tmp - PermissionError: [Errno 13] Permission denied: '/tmp/snap-private-tmp/__init__.py'"
E           and: '=============================== 1 error in 0.44s ==============================='
E       remains unmatched: '*::nbregression(test_nb) FAILED*'

/home/runner/work/pytest-notebook/pytest-notebook/tests/test_plugin_collector.py:42: Failed
----------------------------- Captured stdout call -----------------------------

Workaround

By specifying a local directory to pytest (for example somewhere in /home) with --basetemp=dir avoids running into root-owned dirs triggering that error.

The text was updated successfully, but these errors were encountered:

bluetech · 2024-05-12T18:53:54Z

Thanks for the report, we have a few issues about this already, so I'll close this one.

bluetech closed this as not planned Won't fix, can't repro, duplicate, stale May 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test collection to broad, walking sideways, causes PermissionError for tests in /tmp subdirs #12294

Test collection to broad, walking sideways, causes PermissionError for tests in /tmp subdirs #12294

ibressler commented May 7, 2024 •

edited

bluetech commented May 12, 2024

Test collection to broad, walking sideways, causes PermissionError for tests in /tmp subdirs #12294

Test collection to broad, walking sideways, causes PermissionError for tests in /tmp subdirs #12294

Comments

ibressler commented May 7, 2024 • edited

Workaround

bluetech commented May 12, 2024

ibressler commented May 7, 2024 •

edited