Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package scoped fixture will not execute teardown #8189

Open
hennadii-demchenko opened this issue Dec 23, 2020 · 20 comments
Open

Package scoped fixture will not execute teardown #8189

hennadii-demchenko opened this issue Dec 23, 2020 · 20 comments
Labels
status: help wanted developers would like help from experts on this topic topic: fixtures anything involving fixtures directly or indirectly type: bug problem that needs to be addressed

Comments

@hennadii-demchenko
Copy link

hennadii-demchenko commented Dec 23, 2020

I am experiencing a weird issue:
'package' scoped fixture will not execute teardown after last test in package is completed if afformentioned fixture is defined outside of package

Environment

~ python --version
Python 3.8.5

~ pip freeze
allure-pytest==2.8.24
allure-python-commons==2.8.24
assertpy==1.1
attrdict==2.0.1
attrs==20.3.0
bcrypt==3.2.0
cffi==1.14.4
cryptography==3.2.1
cycler==0.10.0
future==0.18.2
iniconfig==1.1.1
kiwisolver==1.3.1
matplotlib==3.3.3
mininet==2.3.0.dev6
more-itertools==8.6.0
numpy==1.19.4
packaging==20.7
paramiko==2.7.2
pexpect==4.8.0
pexpect-serial==0.1.0
Pillow==8.0.1
pluggy==0.13.1
ptyprocess==0.6.0
py==1.9.0
pycparser==2.20
PyNaCl==1.4.0
pyparsing==2.4.7
Pypubsub==4.0.3
pyserial==3.5
pytest==6.2.1
pytest-dependency==0.5.1
pytest-logger==0.5.1
pytest-sugar==0.9.4
python-dateutil==2.8.1
PyYAML==5.3.1
singleton-decorator==1.0.0
six==1.15.0
termcolor==1.1.0
toml==0.10.2
wcwidth==0.2.5

~ uname -a 
5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Example to reproduce

~ tree test
test
├── __init__.py
├── conftest.py
├── sub1
│   └── __init__.py
│   └── test_foo.py
└── sub2
    └── __init__.py
    └── test_bar.py
2 directories, 3 files

~ cat test/conftest.py
import pytest

@pytest.fixture(scope='package')
def package_scoped():
    print('setup')
    yield
    print('teardown')

~ cat test/sub1/test_foo.py
def test_foo(package_scoped): pass

~ cat test/sub2/test_bar.py
def test_foo(package_scoped): pass

Result

~ pytest test --setup-plan -p no:sugar
=============== test session starts ===============
platform linux -- Python 3.8.5, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/hdemchenko/git/uns
plugins: dependency-0.5.1, logger-0.5.1, allure-pytest-2.8.24
collected 2 items

test/sub1/test_foo.py
  SETUP    P package_scoped
        test/sub1/test_foo.py::test_foo (fixtures used: package_scoped)
test/sub2/test_bar.py
        test/sub2/test_bar.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped

=============== no tests ran in 0.01s ===============

If I create two conftest files in each package with the package scoped fixture I intend to use then I can see it behaving as I was expecting.
Example:

~ mv test/conftest.py test/sub1/
~ cp test/sub1/conftest.py test/sub2/
~ pytest test --setup-plan -p no:sugar
=============== test session starts ===============
platform linux -- Python 3.8.5, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/hdemchenko/git/uns
plugins: dependency-0.5.1, logger-0.5.1, allure-pytest-2.8.24
collected 2 items

test/sub1/test_foo.py
  SETUP    P package_scoped
        test/sub1/test_foo.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped
test/sub2/test_bar.py
  SETUP    P package_scoped
        test/sub2/test_bar.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped

=============== no tests ran in 0.01s ===============
@Zac-HD Zac-HD added status: help wanted developers would like help from experts on this topic topic: fixtures anything involving fixtures directly or indirectly labels Dec 26, 2020
@bluetech
Copy link
Member

Just to make sure I understand, what you expect is that the first example have the same behavior as the second?

You have 3 packages in play: test, test.sub1, test.sub2. In the first example the fixture's scope is package test. In the second example the fixtures' scopes are test.sub1 and test.sub2.

@hennadii-demchenko
Copy link
Author

hennadii-demchenko commented Dec 27, 2020

@Zac-HD, so should I understand package fixture scope such as:

  • starts before the first test requiring the fixture
  • ends with the last test within package scope
  • package scope is literally the package fixture was defined in

To be honest I am a little confuesd with your explanation because every other scope, except of maybe session has different behavior.
For example what should happen if:

  • there is a single module scoped fixture in conftest file at test-root directory named say 'my_fixture'
  • there is some amount of packages in same test-root directory
  • there is some amount of test modules in each package
  • there are some tests in each module and each of them require the same 'my_fixture' fixture

What would setup/teardown plan look like in this case? And to answer your question about my expectation - i kinda expect the same setup/teardown behavior to be applied to package level scopes, with exception that the boundary is on the package but not the module.

@hennadii-demchenko
Copy link
Author

And to proove my point further - another way to get expected (at least by me) behavior is:

  • have test directory structure
~ tree test
test
├── __init__.py
├── conftest.py
├── sub1
│   └── __init__.py
│   └── conftest.py
│   └── test_foo.py
└── sub2
    └── __init__.py
    └── conftest.py
    └── test_bar.py
  • leave test/conftest.py the same as in report
  • have test/sub1/conftest.py and test/sub2/conftest.py simply import fixture and nothing else

@hennadii-demchenko
Copy link
Author

@Zac-HD I think the best way to summarize is that currently package scope is related to "the package" when I expect it to be for "a package"

@bluetech
Copy link
Member

Thanks for the details @hennadii-demchenko. I agree with your intuition here. I think the package scope and/or conftest handling is quite buggy. There's also some related discussion in #7777 (in the sense that it affects the behavior here).

I'm marking this as a bug. Personally I do plan to dig into this more soon.

@bluetech bluetech added the type: bug problem that needs to be addressed label Dec 29, 2020
@hennadii-demchenko
Copy link
Author

@bluetech, thanks I'll keep an eye on the progress.

Here's a quick tip on workaround for other people who might stumble upon this issue (it is easy although not very beatiful)
All you need to do is:

  • create conftest file in each package you want to use the fixture
  • just import the fixture function in conftest file

@hennadii-demchenko
Copy link
Author

@bluetech any luck on getting back to this issue?

@RonnyPfannschmidt
Copy link
Member

to add extra context, each import of a fixture creates a new definition point, thus iporting the fixtures to closer file location points fixes the issues by creating new definitions

its not clear to me whats the exact scoping of package scope fixtures, as they can have multiple points of reference its something tricky to get right

@hennadii-demchenko
Copy link
Author

Hey, it's been a while. Any chance this is gonna be fixed at all?

@stormi
Copy link

stormi commented Apr 26, 2022

I just met this exact issue.

stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue Apr 26, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move common package-scoped fixtures to
  pkgfixtures.py and import them explicitly in test packages'
  conftest.py or directly in test modules.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue Apr 26, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move common package-scoped fixtures to
  pkgfixtures.py and import them explicitly in test packages'
  conftest.py or directly in test modules.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue Apr 27, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move common package-scoped fixtures to
  pkgfixtures.py and import them explicitly in test packages'
  conftest.py or directly in test modules.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue Apr 27, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move shared package-scoped fixtures to
  pkgfixtures.py and import them explicitly in the test packages'
  conftest.py.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue Apr 27, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move shared package-scoped fixtures to
  pkgfixtures.py and import them explicitly in the test packages'
  conftest.py.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189

Signed-off-by: Samuel Verschelde <stormi-xcp@ylix.fr>
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue May 2, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move shared package-scoped fixtures to
  pkgfixtures.py and import them explicitly in the test packages'
  conftest.py.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189

Signed-off-by: Samuel Verschelde <stormi-xcp@ylix.fr>
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue May 3, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move shared package-scoped fixtures to
  pkgfixtures.py and import them explicitly in the test packages'
  conftest.py.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189

Signed-off-by: Samuel Verschelde <stormi-xcp@ylix.fr>
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue May 3, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move shared package-scoped fixtures to
  pkgfixtures.py and import them explicitly in the test packages'
  conftest.py.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189

Signed-off-by: Samuel Verschelde <stormi-xcp@ylix.fr>
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue May 3, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move shared package-scoped fixtures to
  pkgfixtures.py and import them explicitly in the test packages'
  conftest.py.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189

Signed-off-by: Samuel Verschelde <stormi-xcp@ylix.fr>
stormi added a commit to xcp-ng/xcp-ng-tests that referenced this issue May 3, 2022
* Fixtures defined in a test package's conftest.py should not be
  session-scoped. Change the scope to "package".
* Due to a bug in pytest [1] regarding when package-scoped fixtures are
  setup and teared down, move shared package-scoped fixtures to
  pkgfixtures.py and import them explicitly in the test packages'
  conftest.py.
* Add __init__.py to all test packages and to lib/, to make them proper
  python packages.

[1] pytest-dev/pytest#8189

Signed-off-by: Samuel Verschelde <stormi-xcp@ylix.fr>
@akakakakakaa
Copy link

+If uses package scope fixture with autouse=True, It works like session scope. We must use import the fixture function in conftest file.

@bluetech
Copy link
Member

Marking as fixed in pytest 8.0.0 (#11646), let me know if not.

@hennadii-demchenko
Copy link
Author

The issue is still present. @bluetech what should I do with the new findings, should I open a new issue, or shall we reopen this one?

Environment

(.venv) $ pip freeze
iniconfig==2.0.0
packaging==23.2
pluggy==1.4.0
pytest==8.0.1

(.venv) $ python --version
Python 3.11.7

(.venv) $ uname -a
Linux KBP1-LHP-A05452 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

(.venv) $ tree test
test
├── conftest.py
├── __init__.py
├── sub1
│   ├── __init__.py
│   └── test_foo.py
└── sub2
    ├── __init__.py
    └── test_bar.py

(.venv) $ cat conftest.py
import pytest

@pytest.fixture(scope='package')
def package_scoped():
    print('setup')
    yield
    print('teardown')

(.venv) $ cat test/sub1/test_foo.py
def test_foo(package_scoped): pass

(.venv) $ cat test/sub2/test_foo.py
def test_foo(package_scoped): pass

Recreating original steps to reproduce

(.venv) $ pytest test --setup-plan
=========================== test session starts ============================
platform linux -- Python 3.11.7, pytest-8.0.1, pluggy-1.4.0
rootdir: /dev/shm
collected 2 items

test/sub1/test_foo.py
  SETUP    P package_scoped
        test/sub1/test_foo.py::test_foo (fixtures used: package_scoped)
test/sub2/test_bar.py
        test/sub2/test_bar.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped

========================== no tests ran in 0.01s ===========================

Result after adding additional conftest.py file only to the sub1 or both to sub1 and sub2 directories with import of the package fixture

(.venv) $ cat test/sub1/conftest.py
from test.conftest import package_scoped

(.venv) $ pytest test --setup-plan
=========================== test session starts ============================
platform linux -- Python 3.11.7, pytest-8.0.1, pluggy-1.4.0
rootdir: /dev/shm
collected 2 items

test/sub1/test_foo.py
  SETUP    P package_scoped
        test/sub1/test_foo.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped
test/sub2/test_bar.py
  SETUP    P package_scoped
        test/sub2/test_bar.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped

Result after adding additional conftest.py file only to the sub2 directory(just moved it) with import of the package fixture
This one is really weird

(.venv) $ pytest test --setup-plan
=========================== test session starts ============================
platform linux -- Python 3.11.7, pytest-8.0.1, pluggy-1.4.0
rootdir: /dev/shm
collected 2 items

test/sub1/test_foo.py
  SETUP    P package_scoped
        test/sub1/test_foo.py::test_foo (fixtures used: package_scoped)
test/sub2/test_bar.py
  SETUP    P package_scoped
        test/sub2/test_bar.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped
  TEARDOWN P package_scoped

========================== no tests ran in 0.01s ===========================

@bluetech
Copy link
Member

@hennadii-demchenko Apologies for the repetitiveness on my part (I think you probably answered this in one of the previous comments), but can you describe again what you expect to happen in each case above?

@hennadii-demchenko
Copy link
Author

@bluetech, sure, I expect that the package fixture has setup executed at the beginning of each package, and teardown at the end of each package.
Example:

(.venv) $ pytest test --setup-plan
=========================== test session starts ============================
platform linux -- Python 3.11.7, pytest-8.0.1, pluggy-1.4.0
rootdir: /dev/shm
collected 2 items

test/sub1/test_foo.py
  SETUP    P package_scoped
        test/sub1/test_foo.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped
test/sub2/test_bar.py
  SETUP    P package_scoped
        test/sub2/test_bar.py::test_foo (fixtures used: package_scoped)
  TEARDOWN P package_scoped

@hennadii-demchenko
Copy link
Author

@bluetech, can I also ask what was the intended way it should work? I didn't find anything about it in the #7777

@hennadii-demchenko
Copy link
Author

hennadii-demchenko commented Feb 20, 2024

Ok, I think I finally get where the confusion comes from.

Consider following:

  • there's a package "papa"
  • there's two sub-packages in it: "alpha" and "bravo"
  • there's a fixture "foxtrot" inside a contest file in the directory of package "papa"

My assumptions are:

  • when I declare only two tests inside of the "alpha" package, both of which are using fixture "foxtrot"
    • there will be setup just before the first "alpha" test
    • there will be teardown just after the second "alpha" test
    • the "foxtrot" fixture doesn't have to be explicitly declared anywhere inside of the "alpha" package
  • when I also declare three tests inside of the "bravo" package, where the fixture "foxtrot" is used only in the second test
    • there will be setup just before the second "bravo" test
    • there will be teardown just after the second "bravo" test
    • the "foxtrot" fixture doesn't have to be explicitly declared anywhere inside of the "bravo" package

What I think is happening:
the pytest finds the declaration of the "foxtrot" fixture and assumes it applies to the boundaries of the "papa" package but not to the sub-packages ("alpha" and "bravo")

@hennadii-demchenko
Copy link
Author

@bluetech, I am pretty sure this issue will be easily forgotten since it is in a closed state. I understand it is better to open a new one and mention this, please confirm or deny.

@bluetech bluetech reopened this Feb 24, 2024
@ahughesattomra
Copy link

I'm seeing something similar, where we have a package fixture defined in conftest and three packages that each use the fixture. I expected the fixture to run three times, once for each package, but it is running only once per session.

Maybe the same as #8712?

@ydirson
Copy link

ydirson commented Jun 10, 2024

There seems to be an obscure edge when the workaround is not properly applied - who knows, could it give some insights to help find a solution to the core issue...

Let's say that the example above is modified so the test in the bravo package gets the fixture by importing the fixture in the test module itself, instead of importing it in bravo/conftest.py (which is non-existent in this case):

test-pytest-pkgconfig (master)$ git ls-tree -r HEAD:
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391	alpha/__init__.py
100644 blob 9c38efcc7709d279136b2f4f0b3d4c27f0589cb8	alpha/conftest.py
100644 blob 71a86e745c7495f5cde764116d3cf95fe063325a	alpha/test.py
100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391	bravo/__init__.py
100644 blob bbc10c110a091de13ea8944be24517e40e8e418b	bravo/test.py
100644 blob a9188659c3a0132ad7b25334a303ecebb4834934	pkgfixtures.py

The setup will appear to work as long as alpha runs before bravo:

test-pytest-pkgconfig (master)$ pytest --setup-plan alpha/test.py bravo/test.py 
===
platform linux -- Python 3.11.2, pytest-7.2.1, pluggy-1.0.0+repack
rootdir: /home/user/tmp/test-pytest-pkgconfig
plugins: order-1.0.1, dependency-0.5.1
collected 2 items

alpha/test.py 
  SETUP    P foxtrot
        alpha/test.py::test_alpha (fixtures used: foxtrot)
  TEARDOWN P foxtrot
bravo/test.py 
  SETUP    P foxtrot
        bravo/test.py::test_bravo (fixtures used: foxtrot)
  TEARDOWN P foxtrot

... but as soon as for some reason bravo runs first, pytest gots some extra confusion, nesting the feature inside itself, which can cause unexpected behaviour depending on what the feature does:

test-pytest-pkgconfig (master)$ pytest --setup-plan bravo/test.py  alpha/test.py
===
platform linux -- Python 3.11.2, pytest-7.2.1, pluggy-1.0.0+repack
rootdir: /home/user/tmp/test-pytest-pkgconfig
plugins: order-1.0.1, dependency-0.5.1
collected 2 items

bravo/test.py 
  SETUP    P foxtrot
        bravo/test.py::test_bravo (fixtures used: foxtrot)
alpha/test.py 
  SETUP    P foxtrot
        alpha/test.py::test_alpha (fixtures used: foxtrot)
  TEARDOWN P foxtrot
  TEARDOWN P foxtrot

ydirson added a commit to xcp-ng/xcp-ng-tests that referenced this issue Jun 10, 2024
pkgfixture stuff must *always* be used from conftest.py, or else
strange problems appear because of the result shown in
pytest-dev/pytest#8189 (comment)

For this case it results in a test failure when test ordering changes,
because of a safeguard added to the fixture.  And "stop using a test
parameter for sr_disk" commit does cause such a change in test
ordering.

Signed-off-by: Yann Dirson <yann.dirson@vates.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: help wanted developers would like help from experts on this topic topic: fixtures anything involving fixtures directly or indirectly type: bug problem that needs to be addressed
Projects
None yet
Development

No branches or pull requests

8 participants