xfail and skip: What to do with tests you know will fail

<h1 style="font-size: 3em"><tt>xfail</tt> and <tt>skip</tt></h1>
<h2 style="font-size: 2.5em">What to do with tests you know will fail</h2>
 
 
 

Paul Ganssle

 
 
<img src="images/pganssle-logos.svg" height="40px" alt="@pganssle">
 
 
This talk on Github:
<a href="https://github.com/pganssle-talks/pytexas-2022-xfail">pganssle-talks/pytexas-2022-xfail</a>

 
<a rel="license" href="https://creativecommons.org/publicdomain/zero/1.0/">
 <img src="external-images/logos/cc-zero.svg" height="45px">
</a>

Notes:

Hi everyone, welcome, welcome! I'm so excited to be speaking here today! This is my first in-person talk in over 2 years, hopefully I still remember how to do this.

I'm Paul Ganssle, you may know me from some of my open source work — I am a Python core dev, where I usually work on `datetime` related things. In my day job, I'm on a machine learning team at Google, which has a beautiful office just a few blocks away from here.

But today I'm here to talk to you about testing. Specifically about how and why you would want to add tests to your test suite that you know are going to fail.

# `skipif` vs `xfail`

## `skipif` — Use for tests that *are supposed to fail*.

```python
@pytest.mark.skipif(sys.version_info < (3, 8),
 reason="Module-level __getattr__ introduced in Python 3.8")
def test_module_getattr_works():
 with clear_dateutil_imports():
 import dateutil
 assert dateutil.tz is not None # Lazy-loaded
```

## `xfail` — Use for tests that *currently fail, but shouldn't*.

```python
@pytest.mark.xfail(reason="Fractional hours and minutes not implemented yet!")
def test_fractional_hour():
    # ISO 8601 allows fractional hours and minutes
    assert (dateutil.parser.isoparse("2021-03-26T14.5") ==
            datetime.datetime(2021, 3, 26, 14, 30))
```

Notes:

I thought I'd start by clearing up a very common source of confusion, which is the difference between `xfail` and `skip`. Both `skip` and `xfail` are used to indicate that a test isn't going to succeed, but there are both behavioral and semantic differences between these.

The behavioral difference is easy to explain — with `skip` the test is skipped and never run, whereas with `xfail` — which means expected failure — the default is to run the test, but to not consider the entire test suite as failing if only expected failure tests are failing.

The semantic difference between these is a bit subtler — you should use `skipif` if there are conditions under which the test is *supposed* to fail — for example in `dateutil` we support lazy-loaded modules starting in Python 3.8, but we support earlier versions of Python, so we want to skip the lazy-loaded module tests when testing against earlier versions.

With `xfail`, you are indicating that the test *should* be passing, but you haven't gotten around to fixing the bug or implementing the feature yet. For example, the `dateutil` ISO 8601 parsers is supposed to handle all valid ISO formats, but currently it doesn't work for particularly obscure ones like formats with fractional hours. Luckily, no one cares about these formats, so it's not a high priority, but I at some point I do plan to get around to implementing the feature, so hopefully one day this test will start passing.

*From [`dateutil` issue #487](https://github.com/dateutil/dateutil/issues/487#issuecomment-340643745)*

# Example: Perfect square function

```python
import math

def is_perfect_square(n: int) -> bool:
    """Determine if any int i exists such that i × i = n."""
    s = math.sqrt(n)
    return s == int(s)
```

With tests:

```python
import square_mod

import pytest

@pytest.mark.parametrize("n", [0, 1, 2, 4, 9, 16, 25, 36])
def test_squares(n):
    assert square_mod.is_perfect_square(n)

@pytest.mark.parametrize("n", [3, 5, 6, 7, 8, 27, 32])
def test_non_squares(n):
 assert not square_mod.is_perfect_square(n)
```

<pre>
<tt class="hljs">
================================================ test session starts ================================================
platform linux -- Python 3.10.0, pytest-7.1.1, pluggy-1.0.0
collected 14 items

test_square_mod.py .............. [100%]

================================================ 14 passed in 0.02s =================================================
</tt></pre>

Notes:

So let's imagine that I have this function that determines if a given integer is a perfect square, and some tests for that function, and we'll add some tests for it that show that yes indeed that's what this function does - it gives True for all these perfect squares and False for all these non-square numbers. Perfect work for a Thursday afternoon, let's go ahead and ship it straight to production, no problem!

## Found a bug: Add a test

```python
def test_negative():
 assert not square_mod.is_perfect_square(-4)
```

<pre>
<tt class="hljs">
================================================ test session starts ================================================
platform linux -- Python 3.10.0, pytest-7.1.1, pluggy-1.0.0
collected 15 items

test_square_mod.py ..............F [100%]

def test_negative():
> assert not square_mod.is_perfect_square(-4)

test_square_mod.py:17:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

n = -4

def is_perfect_square(n: int) -> bool:
 """Determine if any int i exists such that i × i = n."""
> s = math.sqrt(n)
E ValueError: math domain error

square_mod.py:5: ValueError
============================================== short test summary info ==============================================
FAILED test_square_mod.py::test_negative - ValueError: math domain error
=========================================== 1 failed, 14 passed in 0.08s ============================================
</tt>
</pre>

## xfail: Tests that are *expected* to fail

```python
@pytest.mark.xfail(reason="Bug #11493: Negative values not supported!")
@pytest.mark.parametrize("n", [-1, -3, -4])
def test_negative(n):
    # When called with a negative value for n, this test raises ValueError!
    assert not square_mod.is_perfect_square(m)
```

Failure is expected, so the tests pass:

<pre>
<tt class="hljs">
================================================ test session starts ================================================
platform linux -- Python 3.10.2, pytest-7.1.1, pluggy-1.0.0
plugins: subtests-0.7.0, runtime-xfail-1.0.3, cov-3.0.0, hypothesis-6.39.4
collected 17 items

test_square_mod.py ..............xxx [100%]

=========================================== 14 passed, 3 xfailed in 0.03s ===========================================
</tt>
</pre>

# `XFAIL` becomes `XPASS`

```python
def is_perfect_square(n: int) -> bool:
    """Determine if any real integer i exists such that i × i = n."""

# Negative numbers are not squares according to the definition of the function
 if n < 0:
 return False

s = math.sqrt(n)
 return s == int(s)
```

<pre><tt class="hljs">
================================================ test session starts ================================================
platform linux -- Python 3.10.0, pytest-7.1.1, pluggy-1.0.0
collected 15 items

test_square_mod.py ..............X [100%]

=========================================== 14 passed, 1 xpassed in 0.02s ===========================================
</tt></pre>

# Treating failure to fail as a failure

```python
@pytest.mark.xfail(strict=True,
 raises=ValueError,
 reason="Bug #11493: Negative values not supported")
@pytest.mark.parametrize("n", [-1, -3, -4])
def test_negative(n):
 assert not square_mod.is_perfect_square(n)
```

test_square_mod.py ..............FFF [100%]

## Make `strict=True` the default:

In `tox.ini` / `pytest.ini`:

```ini
[pytest]
xfail_strict = True
```

In `setup.cfg`:

```ini
[tool:pytest]
xfail_strict = True
```

In `pyproject.toml`:

```toml
[tool.pytest.ini_options]
xfail_struct = True
```

See [the pytest documentation](https://docs.pytest.org/en/latest/reference/customize.html) for the latest options for global configuration, or [the documentation on the `xfail_strict` option](https://docs.pytest.org/en/latest/reference/reference.html#confval-xfail_strict).

## Being specific about *why* the test is failing

<pre>
<tt class="hljs">
================================================ test session starts ================================================
platform linux -- Python 3.10.2, pytest-7.1.1, pluggy-1.0.0
plugins: subtests-0.7.0, runtime-xfail-1.0.3, cov-3.0.0, hypothesis-6.39.4
collected 17 items

test_square_mod.py ..............xxx [100%]

**Actually failing because of `NameError`**

## Being specific about *why* the test is failing

```python
@pytest.mark.xfail(
    raises=ValueError,
    reason="Bug #11493: Negative values not supported!"
    )
@pytest.mark.parametrize("n", [-1, -3, -4])
def test_negative(n):
    # When called with a negative value for n, this test raises ValueError!
    assert not square_mod.is_perfect_square(m)
```

<pre><tt class="hljs">
================================================ test session starts ================================================
platform linux -- Python 3.10.0, pytest-7.1.1, pluggy-1.0.0
collected 17 items

test_square_mod.py ..............FFF [100%]

===================================================== FAILURES ======================================================
_________________________________________________ test_negative[-1] _________________________________________________
test_square_mod.py:21: in test_negative
 assert not square_mod.is_perfect_square(m)
E NameError: name 'm' is not defined
_________________________________________________ test_negative[-3] _________________________________________________
test_square_mod.py:21: in test_negative
 assert not square_mod.is_perfect_square(m)
E NameError: name 'm' is not defined
_________________________________________________ test_negative[-4] _________________________________________________
test_square_mod.py:21: in test_negative
 assert not square_mod.is_perfect_square(m)
E NameError: name 'm' is not defined
============================================== short test summary info ==============================================
FAILED test_square_mod.py::test_negative[-1] - NameError: name 'm' is not defined
FAILED test_square_mod.py::test_negative[-3] - NameError: name 'm' is not defined
FAILED test_square_mod.py::test_negative[-4] - NameError: name 'm' is not defined
=========================================== 3 failed, 14 passed in 0.09s ============================================
</tt></pre>

# Using `xfail` and `skipif`: Markers

Bare decorator (with or without `reason`)

```python
@pytest.mark.xfail
def test_my_failing_function():
    assert my_function(-3) == 2

@pytest.mark.xfail(reason="NaN handling is not working properly yet.")
def test_float_function():
    float_function(float("nan"))

@pytest.mark.skip(reason="Probably shouldn't do unconditional skipping")
def test_this_is_pointless():
    some_random_function()
```

# Using `xfail` and `skipif`: Markers

## Boolean condition

```python
@pytest.mark.xfail(sys.version_info > (3, 10),
                   reason="AST handling changed in Python 3.10")
def test_some_ast_tomfoolery():
    my_ast_tomfoolery_function("path/to/python_file.py")

@pytest.mark.skipif(sys.version_info < (3, 9),
 reason="zoneinfo introduced in Python 3.9")
def test_zoneinfo():
 import zoneinfo
 tz = zoneinfo.ZoneInfo("America/Chicago")

...
```

## Condition string

```python
@pytest.mark.skipif("not hasattr(os, 'fspath')")
def path_normalization():
    assert os.fspath(MyPathClass("a/b/c")) == "a/b/c"
```

*Condition strings are discouraged*

Notes:

When using boolean conditions, the `reason` parameter must be specified. `pytest` also supports specifying skip and xfail conditions with a string, which has some weird scoping rules that you can look up. This is the older way of specifying conditions and it is now discouraged, but the one advantage it has over boolean conditions is that the condition itself can serve as the reported reason for the skip, so it is not required to specify `reason`.

# Using `xfail` and `skipif`: Markers

## The `run` parameter

```python
@pytest.mark.xfail(
    run=False
    reason="Test will segfault if run!"
)
def test_surrogate_characters():
    parsed = datetime.fromisoformat("2022-03-26\ud800)13:15:04")
    expected = datetime(2022, 3, 26, 13, 15, 4)
    assert parsed == expected
```

```python
@pytest.mark.xfail(
 sys.version_info < (3, 9),
 run=False,
 reason="Test segfaults earlier than Python 3.9"
)
def test_conditional_segfault():
 function_that_segfaults_on_38()
```

Notes:

Using `run=False` basically turns an `xfail` into a `skip`, but semantically it may be useful for you to mark that a given test should succeed but doesn't. `run=False` only applies when the test is actually marked as `xfail`, so if you specify a condition, this basically turns the test into `skipif`.

# Using `xfail` and `skipif`: Parameterized tests

```python
@pytest.mark.parametrize("n", [
    1,
    pytest.param(0,
        marks=pytest.mark.xfail(
            raises=ZeroDivisionError,
            reason="Zero not handled correctly"
        )
    ),
    pytest.param(-1,
        marks=pytest.mark.xfail(
        reason="Not working for negative numbers."
        )
    ),
    5,
])
def test_is_euler_number(n):
    assert is_euler_number(n)
```

<pre>
<tt class="hljs">
================================================ test session starts =================================================
platform linux -- Python 3.10.2, pytest-7.1.1, pluggy-1.0.0
collected 4 items

test_euler_numbers.py .xx. [100%]

============================================ 2 passed, 2 xfailed in 0.03s ============================================
</tt>
</pre>

# Using `xfail` and `skip`

```python
@pytest.mark.parametrize("payload", list(
    pathlib.Path.glob("test_data/test_data_*")
))
def test_process_data_files(payload):
    with open(payload, "rb") as f:
        test_data = load_data(f)

if test_data.num_elements > 255:
        # Don't actually use this! See next slide!
        pytest.xfail(reason="bug #2445: Processing of data files with > 255 fails.")
```

Can embed these into functions, context managers and fixtures

```python
@contextlib.contextmanager
def in_local_timezone(tz: str) -> Iterator[None]:
    if sys.platform.startswith("win") or os.eviron.get("TZ_CHANGE_DISALLOWED") == "true":
        pytest.skip("Time zone change not allowed")

old_tz = get_current_tz()
    set_tz(tz)
    time.tzset()
    yield
    set_tz(old_tz)
    time.tzset()

def test_in_new_york():
    # This test is skipped automatically in platforms and environments that
    # don't support changing the time zone.
    with in_local_timezone("America/New_York"):
        assert my_date_function("2021-01-01") > my_date_function("2020-01-01")
```

# Papercut: `pytest.xfail()` is basically skip!

- `pytest.xfail()` stops execution immediately — there is no way to get `XPASS`
- `xfail_strict` has no effect
- [This is deliberate](https://github.com/pytest-dev/pytest/issues/7071)

# Solution: [`pytest-runtime-xfail`](https://github.com/okken/pytest-runtime-xfail)

```python
@pytest.mark.parametrize("payload", list(
    pathlib.Path.glob("test_data/test_data_*")
))
def test_process_data_files(payload, runtime_xfail):
    with open(payload, "rb") as f:
        test_data = load_data(f)

if test_data.num_elements > 255:
        # Don't actually use this! See next slide!
        runtime_xfail(reason="bug #2445: Processing of data files with > 255 fails.")
```

# Papercut: `xfail` with `hypothesis`

```python
import hypothesis
from hypothesis import strategies as st

@hypothesis.given(n=st.integers())
def test_my_function(n: int) -> None:
 if n < 0:
 pytest.xfail("Zero and negative numbers not working!")

assert my_function(n) > n
```

## Problems:

- Reports a single top-level `XFAIL`:

<pre>
<tt class="hljs">
================================================ test session starts =================================================
platform linux -- Python 3.10.2, pytest-7.1.1, pluggy-1.0.0
plugins: hypothesis-6.39.4
collected 1 item

test_hypothesis.py x [100%]

================================================= 1 xfailed in 0.22s =================================================
</tt>
</pre>

- Disguises real failures in non-`xfail` cases
- Doesn't work with `pytest-runtime-xfail`

# Using `unittest`

```python
import unittest

class MyTest(unittest.TestCase):
    def test_pass(self):
        self.assertTrue(True)

@unittest.expectedFailure
    def test_xfail(self):
        self.assertTrue(False)

@unittest.expectedFailure
    def test_xpass(self):
        self.assertTrue(True)

```

<div class="fragment disappearing-fragment nospace-fragment fade-out" data-fragment-index="0">
Run with <tt>unittest</tt>:

<pre>
<tt class="hljs">
$ python -m unittest -v
test_pass (test_unittest.MyTest) ... ok
test_xfail (test_unittest.MyTest) ... expected failure
test_xpass (test_unittest.MyTest) ... unexpected success

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (expected failures=1, unexpected successes=1)
</tt></pre>
</div>

<div class="fragment nospace-fragment fade-in" data-fragment-index="0">
Run with <tt>pytest</tt>:

<pre>
<tt class="hljs">
$ pytest test_unittest.py 
================================================ test session starts =================================================
platform linux -- Python 3.10.2, pytest-7.1.1, pluggy-1.0.0
collected 3 items

test_unittest.py .xF [100%]

====================================================== FAILURES ======================================================
_________________________________________________ MyTest.test_xpass __________________________________________________
Unexpected success
============================================== short test summary info ===============================================
FAILED test_unittest.py::MyTest::test_xpass
======================================= 1 failed, 1 passed, 1 xfailed in 0.04s =======================================
</tt>
</pre>
</div>

# `unittest`: Missing features

- No `strict=False` or `run=False`
- No support for `raises`
- No way to specify a reason
- Conditional `xfail` not built-in:

```python
def conditional_xfail(condition):
    if condition:
        return unittest.expectedFailure
    else:
        return lambda x: x
```

# Skipping with `unittest`

Decorators:

```python
class MySkipTest(unittest.TestCase):
    @unittest.skip("Unconditional skipping")
    def skip_unconditional(self):
        pass

@unittest.skipIf(sys.platform.startswith("win"), "Not supported on Windows")
    def test_nix_only(self):
        ...

@unittest.skipUnless(sys.platform.startswith("win"), "Only supported on Windows")
    def test_windows_only(self):
        ...
```

At runtime:

```python
 def test_three_day_weekend(self):
 if 4 <= datetime.now().weekday() <= 6:
 self.skipTest("This test has negotiated a 3-day weekend.")

...

def test_api_integration(self):
        r = requests.get("https://url.to/some/resource")
        if r.status_code != 200:
            raise unittest.SkipTest("Resource not available")

...
```