• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

I'm seeing inconsistent behavior in how errors affect exit code that seems to be based on whether the error affects the whole file or just one line. I expected that any error would be counted as a failure, resulting in exit code 1, but this seems to not be the case unless the error is for an entire file.

Consequently, build pipelines that should fail sometimes succeed. Devs seeing this assume all of the tests passed and code with bugs can be submitted and deployed

This exits with code 1:

class TestBug
    def test_bug(self) -> None:
        assert True

ERROR tests/unit/server/util/garbo_test.py

This exits with code 0:

class TestBug:
    def test_bug(self) -> None:
        a = fake_var

ERROR tests/unit/server/util/garbo_test.py::TestBug::test_bug - NameError: name 'fake_var' is not defined

Both show as errors on the terminal test summary

pytest version: 8.0.0
os: Mac OS 14.4.1

pip list:
aiohttp 3.9.5
aiosignal 1.3.1
annotated-types 0.6.0
anyio 3.7.1
attrs 23.2.0
beanie 1.25.0
certifi 2023.7.22
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
coverage 7.5.1
cryptography 42.0.5
debugpy 1.8.1
Deprecated 1.2.14
distro 1.8.0
dnspython 2.6.1
exceptiongroup 1.2.0
execnet 2.1.1
fastapi 0.110.0
flake8 6.1.0
frozenlist 1.4.1
googleapis-common-protos 1.63.0
grpcio 1.60.1
grpcio-tools 1.60.1
h11 0.14.0
httpcore 1.0.2
httpx 0.25.2
idna 3.7
importlib-metadata 6.11.0
iniconfig 2.0.0
isort 5.13.2
Jinja2 3.1.4
jsonref 1.1.0
kazoo 2.9.0
lazy-model 0.2.0
Levenshtein 0.25.0
lxml 5.2.0
MarkupSafe 2.1.5
mccabe 0.7.0
mongomock 4.1.2
mongomock_motor 0.0.29
motor 3.4.0
multidict 6.0.5
mypy 1.8.0
mypy-extensions 1.0.0
numpy 1.26.4
openai 1.16.0
opentelemetry-api 1.23.0
opentelemetry-exporter-otlp-proto-common 1.23.0
opentelemetry-exporter-otlp-proto-grpc 1.23.0
opentelemetry-instrumentation 0.44b0
opentelemetry-instrumentation-system-metrics 0.44b0
opentelemetry-propagator-b3 1.23.0
opentelemetry-proto 1.23.0
opentelemetry-sdk 1.23.0
opentelemetry-semantic-conventions 0.44b0
packaging 24.0
pandas 2.2.0
pandas-stubs 2.2.1.240316
pip 24.0
pluggy 1.5.0
protobuf 4.25.0
psutil 5.9.8
pycodestyle 2.11.1
pycparser 2.22
pydantic 2.6.4
pydantic_core 2.16.3
pyflakes 3.1.0
pyformance 0.4
PyJWT 2.8.0
pymongo 4.7.2
pytest 8.0.0
pytest-asyncio 0.23.5
pytest-cov 4.1.0
pytest-html 4.1.1
pytest-metadata 3.1.1
pytest-mock 3.12.0
pytest-timeout 2.2.0
pytest-xdist 3.5.0
python-dateutil 2.9.0.post0
python-dotenv 1.0.0
python-graphql-client 0.4.3
python-json-logger 2.0.7
pytz 2024.1
rapidfuzz 3.9.0
regex 2024.5.10
requests 2.31.0
ruff 0.2.1
sentinels 1.0.0
setuptools 69.2.0
signalfx 1.1.14
six 1.16.0
sniffio 1.3.0
sort-requirements 1.3.0
splunk-opentelemetry 1.18.0
sseclient-py 1.8.0
starlette 0.36.3
thrift 0.16.0
tiktoken 0.5.2
toml 0.10.2
tornado 6.4
tqdm 4.66.4
types-colorama 0.4.15.20240311
types-Deprecated 1.2.9.20240311
types-docutils 0.21.0.20240423
types-openpyxl 3.1.0.20240331
types-protobuf 4.24.0.20240311
types-psutil 5.9.5.20240316
types-pycurl 7.45.2.20240311
types-Pygments 2.17.0.20240310
types-pyOpenSSL 24.0.0.20240311
types-pytz 2024.1.0.20240417
types-regex 2023.12.25.20240311
types-requests 2.31.0.20240311
types-setuptools 69.2.0.20240317
typing_extensions 4.8.0
tzdata 2024.1
urllib3 2.1.0
uuid 1.30
uvicorn 0.29.0
websockets 12.0
wrapt 1.16.0
ws4py 0.5.1
yarl 1.9.4
zipp 3.18.1