Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reordering tests with pytest_collection_modifyitems hook fails #12305

Open
michalkret opened this issue May 10, 2024 · 1 comment
Open

Reordering tests with pytest_collection_modifyitems hook fails #12305

michalkret opened this issue May 10, 2024 · 1 comment
Labels
status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity

Comments

@michalkret
Copy link

Hiya pytest folks! I've got the problem with pytest hooks. The checklist was checked as below:

  • [✅] a detailed description of the bug or problem you are having
  • [✅] output of pip list from the virtual environment you are using
  • [✅] pytest and operating system versions
  • [✅] minimal example if possible

pytest version: 8.00
python version: 3.12.2
OS: python:dev-3.12-bookworm Debian container

Issue:
I'm trying to reorder execution reorder of my tests using pytest_collection_modifyitems hook as described in:
https://docs.pytest.org/en/latest/reference/reference.html#pytest.hookspec.pytest_collection_modifyitems

I added the following code in the top dir conftest.py:

def pytest_collection_modifyitems(config, items):
    grouped_items = {}
    for item in items:
        #  marks based reordering logic, not important for the issue
        grouped_items[key].append(item)

    new_items = []
    for key, items in sorted(grouped_items.items()):
        new_items.extend(items)

    items[:] = new_items

I noticed that tests are not running in the expected order, so I added logging code at the end in the modified hook to double check if it's correct:

    for item in items:
        marks = item.iter_markers()
        test_name = item.name
        marks_str = ", ".join(mark.name for mark in marks)
        logging.info(f"Test '{test_name}' has marks: {marks_str}")

and indeed it is. If I however log items from session.items in pytest_collection_finish(session: pytest.Session) hook my order will not be there anymore.
Interestingly enough, moving reorgering logic to pytest_collection_finish hook, solves the problem, though it feels like a hacky workaround.

I see 3 potential causes the the issue in hand:

  • I misunderstood the feature;
  • pytest docs are wrong;
  • pytest_collection_modifyitems has a regression.

I'd be grateful for the help with this one.

pip list:

Package                   Version         Editable project location
------------------------- --------------- ---------------------------
allure-pytest             2.13.2
allure-python-commons     2.13.2
ansicolors                1.1.8
attrs                     23.2.0
autobahn                  23.6.2
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        3.3.2
cryptography              42.0.5
hyperlink                 21.0.0
idna                      3.7
iniconfig                 2.0.0
isodate                   0.6.1
Jinja2                    3.1.3
jsonschema                4.21.1
jsonschema-path           0.3.2
jsonschema-specifications 2023.12.1
labgrid                   23.1a2
lazy-object-proxy         1.10.0
MarkupSafe                2.1.5
more-itertools            10.2.0
mypy                      1.8.0
mypy-extensions           1.0.0
openapi-core              0.19.0
openapi-schema-validator  0.6.2
openapi-spec-validator    0.7.1
packaging                 23.2
parse                     1.20.1
pathable                  0.4.3
pexpect                   4.9.0
pip                       24.0
pluggy                    1.4.0
ptyprocess                0.7.0
pycparser                 2.21
pyserial-labgrid          3.5.0.2
pytest                    8.0.0
pyudev                    0.24.1
pyusb                     1.2.1
PyYAML                    6.0.1
referencing               0.31.1
requests                  2.31.0
rfc3339-validator         0.1.4
rpds-py                   0.18.0
ruff                      0.2.1
setuptools                69.1.1
six                       1.16.0
txaio                     23.1.1
types-PyYAML              6.0.12.20240311
types-requests            2.31.0.20240311
typing_extensions         4.9.0
urllib3                   2.2.0
Werkzeug                  3.0.1
xmodem                    0.4.7
@bluetech
Copy link
Member

It's possible some other plugin reorders as well? Try like this:

@pytest.hookimpl(trylast=True)
def pytest_collection_modifyitems(config, items):
   ...

@nicoddemus nicoddemus added the status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity label May 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nicoddemus @bluetech @michalkret