Parametrize your pytest tests

2023-02-27 · [Tomás Farías Santana]

The two features pytest features I use the most are fixtures and parametrize (pytest.mark.parametrize). This post is about how you can use the latter to write concise, reusable, and readable tests.

More tests in less lines of code with parametrize

The pytest.mark.parametrize decorator generates a test for each tuple of parameters we pass it, which drastically reduces the lines of code in written unit tests, as the same unit test can be re-used for each tuple of different parameters.

Here is an example:

 1import typing
 2
 3import pytest
 4
 5
 6def fibonacci_range(start: int, stop: int, step: int) -> typing.Iterator[int]:
 7    """Like the built-in range, but yields elements of the Fibonacci sequence."""
 8    first = 0
 9    second = 1
10    range_generator = (index for index in range(start, stop, step))
11    next_index = next(range_generator)
12
13    for iteration in range(stop):
14        if iteration == next_index:
15            yield first
16
17            try:
18                next_index = next(range_generator)
19            except StopIteration:
20                break
21
22        first, second = second, first + second
23
24
25TEST_SEQUENCES = {
26    (0, 1, 1): [0],
27    (0, 1, 1): [0],
28    (0, 1, 2): [0],
29    (0, 2, 1): [0, 1],
30    (0, 2, 2): [0],
31    (0, 5, 1): [0, 1, 1, 2, 3],
32    (0, 5, 2): [0, 1, 3],
33    (0, 15, 1): [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377],
34    (0, 15, 2): [0, 1, 3, 8, 21, 55, 144, 377],
35}
36
37
38@pytest.mark.parametrize(
39    "start,stop,step",
40    [
41        (0, 1, 1),
42        (0, 1, 2),
43        (0, 2, 1),
44        (0, 2, 2),
45        (0, 5, 1),
46        (0, 5, 2),
47        (0, 15, 1),
48        (0, 15, 2),
49    ],
50)
51def test_fibonacci_range(start: int, stop: int, step: int):
52    """Test the fibonacci_range function with multiple inputs."""
53    computed_sequence = list(fibonacci_range(start, stop, step))
54    assert computed_sequence == TEST_SEQUENCES[(start, stop, step)]

Code Snippet 1: One unit test turns into 8 with parametrize

Pytest will now generate one test per each tuple we have passed to the pytest.mark.parametrize. The values in each tuple will be used as arguments for the test_fibonacci_range test function. Running pytest will report that our test run included 6 tests, one for each tuple¹:

$  pytest test.py::test_fibonacci_range -vv
============================================ test session starts ==========================================
platform linux -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0 -- $HUGO_BLOG/.direnv/python-3.11.4/bin/python
cachedir: .pytest_cache
rootdir: $HUGO_BLOG/content/articles/parametrize-your-pytest-tests
collected 8 items

test.py::test_fibonacci_range[0-1-1] PASSED                                                           [ 12%]
test.py::test_fibonacci_range[0-1-2] PASSED                                                           [ 25%]
test.py::test_fibonacci_range[0-2-1] PASSED                                                           [ 37%]
test.py::test_fibonacci_range[0-2-2] PASSED                                                           [ 50%]
test.py::test_fibonacci_range[0-5-1] PASSED                                                           [ 62%]
test.py::test_fibonacci_range[0-5-2] PASSED                                                           [ 75%]
test.py::test_fibonacci_range[0-15-1] PASSED                                                          [ 87%]
test.py::test_fibonacci_range[0-15-2] PASSED                                                          [100%]

============================================= 8 passed in 0.01s ============================================

We got 6 tests for the price (in lines of code) of one²!

Next to each item, enclosed in square brackets, we see the parameters used for each test, separated by a dash: [0-1-1], [0-1-2], and so on. Each generated test can also be ran individually by specifying the parameters at the end of the test we wish to run³:

$  pytest 'test.py::test_fibonacci_range[0-1-1]' -vv
============================================ test session starts ==========================================
platform linux -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0 -- $HUGO_BLOG/.direnv/python-3.11.4/bin/python
cachedir: .pytest_cache
rootdir: $HUGO_BLOG/content/articles/parametrize-your-pytest-tests
collected 1 item

test.py::test_fibonacci_range[0-1-1] PASSED                                                          [100%]

============================================= 1 passed in 0.00s ===========================================

Stacking parametrize decorators

In our example, we passed multiple arguments to parametrize test_fibonacci_range: start, stop, and step. Writing down all the possible parameter tuples we are interested in testing for each argument combination can take up a lot of time. Thankfully, pytest allows us to stack pytest.mark.parametrize decorators to get all possible combinations of parameters.

By stacking decorators, our example test can be re-written as:

1@pytest.mark.parametrize("start", [0])
2@pytest.mark.parametrize("stop", [1, 2, 5, 15])
3@pytest.mark.parametrize("step", [1, 2])
4def test_fibonacci_range_stacked(start: int, stop: int, step: int):
5    """Test the fibonacci_range function with multiple inputs."""
6    computed_sequence = list(fibonacci_range(start, stop, step))
7    assert computed_sequence == TEST_SEQUENCES[(start, stop, step)]

Code Snippet 2: Stacking parametrize decorators produces a Cartesian product of all parameters

Which generates the same 8 parameter tuples passed to parametrize the test as the previous version of this example:

$  pytest test.py::test_fibonacci_range_stacked -vv
============================================= test session starts =========================================
platform linux -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0 -- $HUGO_BLOG/.direnv/python-3.11.4/bin/python
cachedir: .pytest_cache
rootdir: $HUGO_BLOG/content/articles/parametrize-your-pytest-tests
collected 8 items

test.py::test_fibonacci_range_stacked[1-1-0] PASSED                                                 [ 12%]
test.py::test_fibonacci_range_stacked[1-2-0] PASSED                                                 [ 25%]
test.py::test_fibonacci_range_stacked[1-5-0] PASSED                                                 [ 37%]
test.py::test_fibonacci_range_stacked[1-15-0] PASSED                                                [ 50%]
test.py::test_fibonacci_range_stacked[2-1-0] PASSED                                                 [ 62%]
test.py::test_fibonacci_range_stacked[2-2-0] PASSED                                                 [ 75%]
test.py::test_fibonacci_range_stacked[2-5-0] PASSED                                                 [ 87%]
test.py::test_fibonacci_range_stacked[2-15-0] PASSED                                                [100%]

============================================= 8 passed in 0.01s ===========================================

When generating parameter tuples, pytest will iterate over all the parameters in one decorator before advancing to the next parameter in the decorators that follow.

This is equivalent to a for loop:
for step in [1, 2]:
    for stop in [1, 2, 5, 15]:
        for start in [0]:
            yield (step, stop, start)

Pass parameters to pytest fixtures

Pytest fixtures are used to hide away complicated setup steps required for unit testing. We can pass parameters to fixtures by matching the name of a fixture with the name of an argument used in pytest.mark.parametrize and setting the indirect argument.

For example, imagine you build an app that can interact with multiple database backends. Regardless, of the database backend in use, our app should function the same. As we have abstracted all database interaction under a common interface, we can write a single test and parametrize it to run with multiple backends. We already have fixtures that return test clients for each of our databases, used in the unit tests for each of the clients, so we can write a fixture that can returns both according to how we parametrize it:

 1import pytest
 2
 3
 4@pytest.fixture
 5def db_client(request, postgres_client, mysql_client):
 6    if request.param == "postgres":
 7        return postgres_client
 8    elif request.param == "mysql":
 9        return mysql_client
10    else:
11        raise ValueError(f"Unsupported db: '{request.param}'")
12
13
14@pytest.mark.parametrize("db_client", ["postgres", "mysql"], indirect=True)
15def test_db_operation(db_client):
16    """Test an operation that can be executed on multiple RDBMS."""
17    ...

Any application test can use the db_client fixture and ensure it behaves the same regardless of database backend.

Customize the parameter ids in the test report

Pytest allows us to customize the id of each parameter that will be shown in the test report. This can be useful to have human readable names in our test reports.

Coming back to our first example:

1@pytest.mark.parametrize("start", [0])
2@pytest.mark.parametrize("stop", [1, 2, 5, 15])
3@pytest.mark.parametrize("step", [1, 2], ids=["single", "double"])
4def test_fibonacci_range_with_ids(start: int, stop: int, step: int):
5    """Test the fibonacci_range function with multiple inputs."""
6    computed_sequence = list(fibonacci_range(start, stop, step))
7    assert computed_sequence == TEST_SEQUENCES[(start, stop, step)]

Code Snippet 3: Our step arguments now have names

In our test report, pytest will output “single” and “double” as the parameter ids instead of 1 and 2 respectively:

$  pytest test.py::test_fibonacci_range_with_ids -vv
============================================= test session starts =========================================
platform linux -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0 -- $HUGO_BLOG/.direnv/python-3.11.4/bin/python
cachedir: .pytest_cache
rootdir: $HUGO_BLOG/content/articles/parametrize-your-pytest-tests
collected 8 items

test.py::test_fibonacci_range_with_ids[single-1-0] PASSED                                           [ 12%]
test.py::test_fibonacci_range_with_ids[single-2-0] PASSED                                           [ 25%]
test.py::test_fibonacci_range_with_ids[single-5-0] PASSED                                           [ 37%]
test.py::test_fibonacci_range_with_ids[single-15-0] PASSED                                          [ 50%]
test.py::test_fibonacci_range_with_ids[double-1-0] PASSED                                           [ 62%]
test.py::test_fibonacci_range_with_ids[double-2-0] PASSED                                           [ 75%]
test.py::test_fibonacci_range_with_ids[double-5-0] PASSED                                           [ 87%]
test.py::test_fibonacci_range_with_ids[double-15-0] PASSED                                          [100%]

====================================================== 8 passed in 0.01s ==================================

This is significantly more useful with complex types where pytest’s default behavior is to take the argument name and concatenate an index, like with instances of datetime.datetime:

 1import datetime as dt
 2
 3import pytest
 4
 5first_day_of_month = [dt.datetime(2023, month, 1) for month in range(1, 13)]
 6
 7
 8@pytest.mark.parametrize("date", first_day_of_month)
 9def test_year_is_2023(date):
10    """Dummy test."""
11    assert date.year == 2023
12
13
14@pytest.mark.parametrize(
15    "date",
16    first_day_of_month,
17    ids=map(lambda d: d.strftime("%B"), first_day_of_month),
18)
19def test_year_is_2023_with_ids(date):
20    """Dummy test."""
21    assert date.year == 2023

Code Snippet 4: Using month names as ids

The pytest test report will show date{index} for test_year_is_2023 and the month names for test_year_is_2023_with_ids:

$  pytest test_2.py -vv
============================================= test session starts =========================================
platform linux -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0 -- $HUGO_BLOG/.direnv/python-3.11.4/bin/python
cachedir: .pytest_cache
rootdir: $HUGO_BLOG/content/articles/parametrize-your-pytest-tests
collected 24 items

test_2.py::test_year_is_2023[date0] PASSED                                                          [  4%]
test_2.py::test_year_is_2023[date1] PASSED                                                          [  8%]
test_2.py::test_year_is_2023[date2] PASSED                                                          [ 12%]
test_2.py::test_year_is_2023[date3] PASSED                                                          [ 16%]
test_2.py::test_year_is_2023[date4] PASSED                                                          [ 20%]
test_2.py::test_year_is_2023[date5] PASSED                                                          [ 25%]
test_2.py::test_year_is_2023[date6] PASSED                                                          [ 29%]
test_2.py::test_year_is_2023[date7] PASSED                                                          [ 33%]
test_2.py::test_year_is_2023[date8] PASSED                                                          [ 37%]
test_2.py::test_year_is_2023[date9] PASSED                                                          [ 41%]
test_2.py::test_year_is_2023[date10] PASSED                                                         [ 45%]
test_2.py::test_year_is_2023[date11] PASSED                                                         [ 50%]
test_2.py::test_year_is_2023_with_ids[January] PASSED                                               [ 54%]
test_2.py::test_year_is_2023_with_ids[February] PASSED                                              [ 58%]
test_2.py::test_year_is_2023_with_ids[March] PASSED                                                 [ 62%]
test_2.py::test_year_is_2023_with_ids[April] PASSED                                                 [ 66%]
test_2.py::test_year_is_2023_with_ids[May] PASSED                                                   [ 70%]
test_2.py::test_year_is_2023_with_ids[June] PASSED                                                  [ 75%]
test_2.py::test_year_is_2023_with_ids[July] PASSED                                                  [ 79%]
test_2.py::test_year_is_2023_with_ids[August] PASSED                                                [ 83%]
test_2.py::test_year_is_2023_with_ids[September] PASSED                                             [ 87%]
test_2.py::test_year_is_2023_with_ids[October] PASSED                                               [ 91%]
test_2.py::test_year_is_2023_with_ids[November] PASSED                                              [ 95%]
test_2.py::test_year_is_2023_with_ids[December] PASSED                                              [100%]

===================================================== 24 passed in 0.02s ==================================

Why not to parametrize

Although pytest.mark.parametrize has become a staple of my unit tests, it comes at the cost of complexity and performance, like with many other abstraction layers.

As pytest.mark.parametrize makes it really easy to generate new tests, it is tempting to want to include as many parameter combinations as possible. This temptation comes up a lot when stacking pytest.mark.parametrize decorators.

But doing so cause problems:

We may be led to believe our tests are exhaustive when in fact we are not covering our problem domain.
- “My unit test is has coverage of every possible 32-bit signed integer, what do you mean it’s failing?”.
- “Well, a user has a balance of $0.50 in their account…”.
With too many tests, a test suite can take too long to run⁴.
- A test suite that nobody runs is useless, and the more time a test suite takes to run, the less frequently it will be ran.
It can be easy to obfuscate the generated test cases by setting (or not setting) ids, or with complex code to generate argument tuples.
- With every new argument we parametrize, the number of possible combinations (and in turn the number of tests) can grow exponentially.
- When debugging, we now may need to keep in our minds not only the test but the code that generates the test.

In conclusion

When I started using pytest I was mostly annoyed about having to replace all my self.assertEqual calls for assert statements. It wasn’t until I started diving into its documentation that I learned why is it so loved as a testing framework. pytest.mark.parametrize is just one of the features of pytest I regularly now employ in my unit tests, and I wanted to give you a glimpse of how that looks.

There is a lot more going for pytest, like fixtures and plugins, and I hope to cover more of that in the future.

I like the expressiveness of 2 levels of verbosity (-vv). ↩︎
And the “price” of writing the decorator, which remains constant relative to the lines in our test. ↩︎
Notice the test item is enclosed in quotes. Alternatively, we would have to escape the square brackets. ↩︎
This point can be addressed by having a reduced test suite to run during development, and a complete test suite to run before deployment. ↩︎

Reply to this post by email ↪