Appearance

Test Result Logging ​

Overview ​

The pytest-f3ts plugin leverages the built-in logging features of the pytest module to generate log files. To create records containing test case information such as measurement values, test limits, and error codes the JUnitXML format is used.

JUnitXML is a popular format that is used by many different software test tools and we have found it to be a great format to use for analyzing test results.

Standard Logging Variables ​

The f3ts plugin utilizes the following variables to display information on the Test Runner application:

  • test_id
  • description
  • error_code
  • error_msg
  • min_limit
  • max_limit
  • meas

pytest Test Script Logging ​

Native Pytest ​

In this case, you simply set variables that provide the set values that are used within the test. For example, to test if a measured voltage is within a given min and max limit, the following could be written:

python
def test_5v0_voltage_rail():
    # Measure voltage
    meas = get_5v0_voltage()

    # Check against a low and high limit
    assert meas > 4.9, "Voltage too low"
    assert meas < 5.1, "Voltage too high"

This test would run perfectly and produce a pass/fail result depending on the measured voltage, however, the generated log files would not contain any information on the measured value or limits that were used during the test.

record_property ​

To add the standard logging values to the JUnit-based XML test log, the record_property fixture can be used:

python
# Add the `record_property` fixture to the test case
def test_5v0_voltage_rail(record_property):
    # Set test limits as variables and add them to the test log
    min_limit = 4.9
    record_property("min_limit", 4.9)
    max_limit = 5.1
    record_property("max_limit", 5.1)

    # Measured voltage and add to the test log
    meas = get_5v0_voltage()
    record_property("meas", meas)

    # Check against a low and high limit
    assert meas > min_limit, "Voltage too low"
    assert meas < max_limit, "Voltage too high"

A utility function called log_vars is available to automatically log the standard configuration variables that are used by the Test runner application.

python
# Import the `log_vars` utility function
from pytest_f3ts.utils import log_vars


# Add the `record_property` fixture to the test case
def test_5v0_voltage_rail(record_property):
    # Set test limits as variables and add them to the test log
    min_limit = 4.9
    max_limit = 5.1

    # Measured voltage and add to the test log
    meas = get_5v0_voltage()

    # Call log_vars prior to any assertions
    log_vars(record_property)

    # Check against a low and high limit
    assert meas > min_limit, "Voltage too low"
    assert meas < max_limit, "Voltage too high"

By adding this call, the max_limit, min_limit, and meas variables are automatically added to the JUnitXML report.

Logging multiple per test ​

f3ts Assertion Wrapper ​

The pytest-f3ts fixture f3ts_assert can be used instead of a typical pytest assert call to run the assertion statement and send the assertion results directly to the Test Executer (TE) as a TestResult object.

Assert statements made with f3ts_assert will show up as sub results within the test application GUI and stored as a result in the results database. This will take in all of the configuration variables as arguments and pass sub-assertions to the Test Executer Directly.

python
def test_multiple_voltages(f3ts_assert):

    measurement = test_get_5V0()

    f3ts_assert(4.5 <= measurement <= 5.5,
                min_limit=4.5, meas=measurement, max_limit=5.5,
                error_msg="5V Outside of Limits")

    measurement = test_get_2V0()

    f3ts_assert(measurement <= 2.5,
                meas=measurement, max_limit=2.5,
                error_msg="2V Outside of Limits")

Log File Output ​

pytest-f3ts records each test's standard variables (test_id, description, error_code, error_msg, min_limit, max_limit, meas) onto the test's user_properties via record_property. Run pytest with --junitxml to write these to a JUnit-based XML report:

bash
pytest -p f3ts --junitxml=results.xml

The plugin does not emit log files in any other format on its own. If you want additional report formats, use the standard pytest reporters that ship as dependencies of pytest-f3ts — for example pytest-html for an HTML report or pytest-reportlog for line-based JSON:

bash
pytest -p f3ts --html=report.html
pytest -p f3ts --report-log=results.jsonl

Offline summary table ​

When no Test Runner backend is connected, the plugin prints a human-readable summary table to the terminal at the end of the run (the "FixturFab Test Runner Summary"). Pass --skip-summary to suppress it:

bash
pytest -p f3ts --skip-summary

Known Issues ​

JUnitXML xunit2 and record_property ​

The record_property method is not supported by xunit2, due to the need to record additional properties, pytest_f3ts always uses xunit. By default, a warning message will be displayed like the following:

bash
 test_plan.py:6: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
    def test_continuity(test_config, record_property):

Whenever the record_property fixture is called. To prevent this warning, add the following to the pyproject.toml file:

toml
[tool.pytest.ini_options]
junit_family = "xunit1"