[Python for DevOps] Pytest for DevOps!
August 10, 2020
Ok, this time I made a jump from chapter 3, to directly chapter 8 in the book Python for DevOps, because I needed to learn about writing tests/unitests in python.
When I say writing tests, I broadly mean writing validation steps, to check if everything is happening the intended way or not. And if not, then what kind of behaviour I’m receiving from my program, or infrastructure.
(Note: The notes below are the direct excerpts from the book, Python For DevOps, compiled for the purpose of learning & memory.)
This validation can happen at every step of the way and when achieving important objectives.
For example, if in the middle of a long list of steps to produce a deployment, a curl command is called to get an all-important file, do you think the build should continue if it fails? Probably not! curl has a flag that can be used to produce a nonzero exit status (–fail) if an HTTP error happens. That simple flag usage is a form of validation: ensure that the request succeeded, otherwise fail the build step.
This chapter talked about:
- the basics associated with testing in Python using the phenomenal
- then dives into some advanced features of the framework.
- finally goes into detail about the TestInfra project, a plug-in to pytest that can do system verification.
pytestis a command-line tool that discovers Python tests and executes them.
- It doesn’t force a user to understand its internals, which makes it easy to get started with.
Testing with Pytest!
- Create a python virtual environment (venv) named
testing. And make sure
pytestis installed & available in the command line.
$ python3 -m venv testing $ source testing/bin/activate $ pip3 install pytest
- Create a file called
test_basic.py. It should look like this:
def test_simple(): assert True def test_fails(): assert False
- If pytest runs without any arguments, it should show a pass and a failure.
(testing) $ pytest ====================== test session starts ======================= platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 rootdir: /home/priyankasaggu119/Desktop/.myhome/python-study plugins: testinfra-2.1.0 collected 2 items test_basic.py .F [100%] ============================ FAILURES ============================ ___________________________ test_fails ___________________________ def test_fails(): > assert False E assert False test_basic.py:5: AssertionError ==================== short test summary info ===================== FAILED test_basic.py::test_fails - assert False ================== 1 failed, 1 passed in 0.13s ===================
- The output is beneficial from the start:
- it displays how many tests were collected.
- how many of them passed.
- and which one failed—including its line number.
The default output from pytest is handy, but it might be too verbose. You can control the amount of output with configuration, reducing it with the
- There was no need to create a class to include the tests; functions were discovered and ran correctly. A test suite can have a mix of both, and the framework works fine in such an environment.
These are conventions that will allow the tool to discover tests:
- The testing directory needs to be named `tests`. - Test files need to be prefixed with `test`. For example, `test_basic.py`, or suffixed with `test.py`. - Test functions need to be prefixed with `test_`, for example, `def testsimple():`. - Test classes need to be prefixed with `Test`; for example, `class TestSimple`. - Test methods follow the same conventions as functions, prefixed with `test_`; for example, `def test_method(self):`.
- Because prefixing with
test_is a requirement for automatic discovery and execution of tests, it allows introducing helper functions and other nontest code with different names, so that they get excluded automatically.
Python already comes with a set of utilities and helpers for testing, and they are part of the
unittest module. It is useful to understand how
pytest is different and why it is highly recommended.
unittestmodule forces the use of classes and class inheritance. For an experienced developer who understands object-oriented programming and class inheritance, this shouldn’t be a problem, but for beginners, it is an obstacle. Using classes and inheritance shouldn’t be a requisite for writing basic tests!
unitest] Part of forcing users to inherit from unittest.TestCase is that you are required to understand (and remember) most of the assertion methods that are used to verify results.
pytest, there is a single assertion helper that can do it all:
pytestallows you to use assert exclusively and does not force you to use any of the above. Moreover, it does allow you to write tests using unittest, and it even executes them. We strongly advise against doing that and suggest you concentrate on just using plain asserts.
pytest] Not only is it easier to use plain asserts, but pytest also provides a rich comparison engine on failures (more on this in the next section).
Hooks are an advanced feature of pytest that you might not need at all, but it is useful to understand that the framework can be flexible enough to accommodate different requirements.
Here, we will talk about how to extend the framework.
- why using
assertis so valuable.
- how to
parametrizetests to reduce repetition.
- how to make helpers with
fixtures, and how to use the built-in ones.
conftest.pyfile is used to hold hooks, fixtures, and helpers for those fixtures. Those fixtures can then be used within tests if declared as arguments.
(I’ll discuss about fixtures later in the blog!)
It makes sense to add fixtures and helpers to this file when more than one test module will use it. If there is only a single test file, or if only one file is going to make use of a fixture or hook, there is no need to create or use a conftest.py file. Fixtures and helpers can be defined within the same file as the test and behave the same.
- The only condition for loading a
conftest.pyfile is to be present in the tests directory and match the name correctly.
Difference between using direct
assert and using pytest
>>> assert "using assert for errors" == "using asert for errors" Traceback (most recent call last): File "<stdin>", line 1, in <module> AssertionError
Here, it is hard to tell the error/issue without spending some time looking at those two long lines closely.
Now let’s look at the same
assert inside a test file.
def test_long_files(): assert "using assert for errors" == "using asert for errors"
(testing) $ pytest test_long_lines.py ====================== test session starts ======================= platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 rootdir: /home/priyankasaggu119/Desktop/.myhome/python-study plugins: testinfra-2.1.0 collected 1 item test_ll.py F [100%] ============================ FAILURES ============================ ________________________ test_long_files _________________________ def test_long_files(): > assert "using assert for errors" == "using asert for errors" E AssertionError: assert 'using assert for errors' == 'using asert for errors' E - using asert for errors E + using assert for errors E ? + test_ll.py:2: AssertionError ==================== short test summary info ===================== FAILED test_ll.py::test_long_files - AssertionError: assert 'us... ======================= 1 failed in 0.12s ========================
This time, telling where the error is? is tremendously easier. Not only does it tell you it fails, but it points to exactly where the failure is.
The example is a simple assert with a long string, but the framework handles other data structures like lists and dictionaries without a problem.
Parametrization is one of the features which become clear once you find yourself writing very similar tests that had minor changes in the inputs but are testing the same thing.
Take, for example, this class that is testing a function that returns
True if a string is implying a truthful value. The
string_to_bool is the function under test.
from my_module import string_to_bool class TestStringToBool(object): def test_it_detects_lowercase_yes(self): assert string_to_bool('yes') def test_it_detects_odd_case_yes(self): assert string_to_bool('YeS') def test_it_detects_uppercase_yes(self): assert string_to_bool('YES') def test_it_detects_positive_str_integers(self): assert string_to_bool('1') def test_it_detects_true(self): assert string_to_bool('true') def test_it_detects_true_with_trailing_spaces(self): assert string_to_bool('true ') def test_it_detects_true_with_leading_spaces(self): assert string_to_bool(' true')
See how all these tests are evaluating the same result from similar inputs?
This is where parametrization shines because it can group all these values and pass them to the test. Thus, it can effectively reduce them to a single test.
import pytest from my_module import string_to_bool true_values = ['yes', '1', 'Yes', 'TRUE', 'TruE', 'True', 'true'] class TestStrToBool(object): @pytest.mark.parametrize('value', true_values) def test_it_detects_truish_strings(self, value) assert string_to_bool(value)
There are a couple of things happening here.
pytestis imported (the framework) to use the
true_valuesis defined as a (list) variable of all the values to use that should evaluate the same.
- finally, it replaces all the test methods to a single one.
- The test method uses the
parametrizedecorator, which defines two arguments.
- first is a string,
- second is the name of the list
true_values, defined previously.
- first is a string,
- Increase the verbosity of the test by adding
-vflag, and the output will include the values used in each iteration of the single test in brackets.
(testing) $ pytest test_something.py -v ================================ test session starts ================================ platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /home/priyankasaggu119/Desktop/.myhome/python-study plugins: testinfra-2.1.0 collected 7 items test_something.py::TestStrToBool::test_it_detects_truish_strings[yes] PASSED [ 14%] test_something.py::TestStrToBool::test_it_detects_truish_strings PASSED [ 28%] test_something.py::TestStrToBool::test_it_detects_truish_strings[Yes] PASSED [ 42%] test_something.py::TestStrToBool::test_it_detects_truish_strings[TRUE] PASSED [ 57%] test_something.py::TestStrToBool::test_it_detects_truish_strings[TruE] PASSED [ 71%] test_something.py::TestStrToBool::test_it_detects_truish_strings[True] PASSED [ 85%] test_something.py::TestStrToBool::test_it_detects_truish_strings[true] PASSED [100%] ================================= 7 passed in 0.04s =================================
(Refer the official docs for fixtures (here.)[https://docs.pytest.org/en/latest/fixture.html])
fixtures are like little helpers that can get injected into a test. Regardless of whether you are writing a single test function or a bunch of test methods, fixtures can be used in the same way.
If they aren’t going to be shared among other test files, it is fine to define them in the same test file; otherwise they can go into the
Fixtures, just like helper functions, can be almost anything you need for a test, from simple data structures that get pre-created to more complex ones like setting a database for a web application.
These helpers can also have a defined scope. They can have specific code that cleans up for every test method, class, and module, or even allows setting them up once for the whole test session. By defining them in a test method (or test function), you are effectively getting the fixture injected at runtime.
Fixtures as Function arguments
- Test functions can receive fixture objects by naming them as an input argument.
- For each argument name, a fixture function with that name provides the fixture object.
- Fixture functions are registered by marking them with
# content of ./test_smtpsimple.py import pytest @pytest.fixture def smtp_connection(): import smtplib return smtplib.SMTP("smtp.gmail.com", 587, timeout=5) def test_ehlo(smtp_connection): response, msg = smtp_connection.ehlo() assert response == 250 assert 0 # for demo purposes
- Here, the
pytestwill discover and call the
(testing) $ pytest test_smtpsimple.py ================================ test session starts ================================ platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 rootdir: /home/priyankasaggu119/Desktop/.myhome/python-study plugins: testinfra-2.1.0 collected 1 item test_smtpsimple.py F [100%] ===================================== FAILURES ====================================== _____________________________________ test_ehlo _____________________________________ smtp_connection = <smtplib.SMTP object at 0x7fe07285e7f0> def test_ehlo(smtp_connection): response, msg = smtp_connection.ehlo() assert response == 250 > assert 0 # for demo purposes E assert 0 test_smtpsimple.py:14: AssertionError ============================== short test summary info ============================== FAILED test_smtpsimple.py::test_ehlo - assert 0 ================================= 1 failed in 0.82s =================================
In the failure traceback, we see that the test function was called with a
smtp_connection argument, the
smtplib.SMTP() instance created by the fixture function. The test function fails on our deliberate
Here is the exact protocol used by
pytest to call the test function this way:
pytest finds the test
test_ehlo because of the
test_ prefix. The test function needs a function argument named
smtp_connection. A matching fixture function is discovered by looking for a fixture-marked function named
smtp_connection() is called to create an instance.
test_ehlo(<smtp_connection instance>) is called and fails in the last line of the test function.
- To verify the full list of available built-in fixtures, run the following command.
$ (testing) pytest -q --fixtures
There are two fixtures that we use a lot:
capsyscaptures any stdout or stderr produced in a test, in a tupe
These are two test functions that verify the output produced on stderr and stdout, respectively.
import sys def stderr_logging(): sys.stderr.write('stderr output being produced') def stdout_logging(): sys.stdout.write('stdout output being produced') def test_verify_stderr(capsys): stderr_logging() out, err = capsys.readouterr() assert out == '' assert err == 'stderr output being produced' def test_verify_stdout(capsys): stdout_logging() out, err = capsys.readouterr() assert out == 'stdout output being produced' assert err == ''
capsysfixture handles all the patching, setup, and helpers to retrieve the
stdoutproduced in the test.
- The content is reset for every test, which ensures that the variables populate with the correct output.
monkeypatchis probably the fixture that we use the most.
When testing, there are situations where the code under test is out of our control, and patching needs to happen to override a module or function to have a specific behavior. There are quite a few patching and mocking libraries (mocks are helpers to set behavior on patched objects) available for Python, but
monkeypatchis good enough that you might not need to install a separate library to help out.
- The following function runs a system command to capture details from a device, then parses the output, and returns a property (the
ID_PART_ENTRY_TYPEas reported by
import subprocess def get_part_entry_type(device): """ Parses the ``ID_PART_ENTRY_TYPE`` from the "low level" (bypasses the cache) output that uses the ``udev`` type of output. """ stdout = subprocess.check_output(['blkid', '-p', '-o', 'udev', device]) for line in stdout.split('\n'): if 'ID_PART_ENTRY_TYPE=' in line: return line.split('=')[-1].strip() return '' def test_parses_id_entry_type(monkeypatch): monkeypatch.setattr( 'subprocess.check_output', lambda cmd: '\nID_PART_ENTRY_TYPE=aaaaa') assert get_part_entry_type('/dev/sda') == 'aaaa'
setattrcall sets the attribute on the patched callable (
check_outputin this case).
- It patches it with a lambda function that returns the one interesting line. Since the
subprocess.check_outputfunction is not under our direct control, and the
get_part_entry_typefunction doesn’t allow any other way to inject the values, patching is the only way.
(testing) $ pytest test_monkeypath.py ================================ test session starts ================================ platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 rootdir: /home/priyankasaggu119/Desktop/.myhome/python-study plugins: testinfra-2.1.0 collected 1 item test_monkeypath.py F [100%] ===================================== FAILURES ====================================== _____________________________ test_parses_id_entry_type _____________________________ monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fc9407b0670> def test_parses_id_entry_type(monkeypatch): monkeypatch.setattr( 'subprocess.check_output', lambda cmd: '\nID_PART_ENTRY_TYPE=aaaaa') > assert get_part_entry_type('/dev/sda') == 'aaaa' E AssertionError: assert 'aaaaa' == 'aaaa' E - aaaa E + aaaaa E ? + test_monkeypath.py:18: AssertionError ============================== short test summary info ============================== FAILED test_monkeypath.py::test_parses_id_entry_type - AssertionError: assert 'aaa... ================================= 1 failed in 0.12s =================================
So, till here, We have discussed all writing basic
pytest tests, then exploring the advanced features provided by the module like helper function hooks,
conftest.py file, parametrization, custom & built-in
fixtures and lots of supporting examples.
I’m breaking it here, & will cover the Infrastructure Testing using
pytest module & plugins, in the next blog!