Univention Bugzilla – Bug 40709
ucs-test: switch to pytest
Last modified: 2024-03-15 10:12:52 CET
pytest is a very nice testing framework for python: http://pytest.org/latest/ It provides e.g. the following features: * multiple test definitions in 1 file → e.g. would be super helpful in UCS@school tests → if one computerroom test fails every other is not executed anymore * use of "assert" statement to fail a test which prints very nice comparison of the assertion AND the traceback (which doesn't happen for univention.testing.utils.fail) * better output + summary * pytest has markers → every function can be marked e.g. with @pytest.mark.only_admember * conditional skipping of tests * XFAIL-ing tests: Tests which are expected to fail (e.g. with a specific exception or string) are executed to see if they still fail or if they fail for another reason or maybe they pass again? This is shown in the summary. → Currently we just skip all tests if they fail to get green results. This would give use also better statistics (to see test are failing, which are skipped and which are working but the results are still green in jenkins). * you can define so called "fixtures" for different scopes → helper functions * let tests run twice if they failed * Tests can get parametrized with different arguments / input * coverage integration → e.g. we can see which codelines aren't covered in UDM handlers after running all tests. Bug #37102 Maybe it's possible to integrate pytest into ucs-test somehow?
This works already: Just define the following hash-bang in your test: #!/usr/share/ucs-test/runner /usr/bin/py.test -s The only drawback is that some fixtures can't be shared among multiple modules/test cases so they are created for each module. Another drawback is that ucs-test / jenkins doesn't show succeeded XFAILing tests.
This still can be improoved. E.g. the output format currently lists all tests in one test section. We should write the junit results and merge them into the others. I found that ucs-test also has somehting like XFAIL / XPASS: exitcode 101: ('O', 'REASON_FIXED_EXPECTED', 'Test passed', 'GREEN'), exitcode 102: ('O', 'REASON_FIXED_UNEXPECTED', 'Test passed', 'GREEN'), Maybe it's possible to let py-test run all ucs-test cases and we map the status codes, conditions, etc.?
FYI: Tests like 61_udm-users/100_test_users.py or 55_adconnector/50*.py are too long running for my taste (some run about 30 minutes) because they iterate over several (combinations of) options. Maybe we can make this more usable by improving the error message to show which py.test command to run to repeat the individual test unit. Otherwise we should break them up into separate files.
(In reply to Arvid Requate from comment #3) > FYI: Tests like 61_udm-users/100_test_users.py or 55_adconnector/50*.py are > too long running for my taste (some run about 30 minutes) because they > iterate over several (combinations of) options. Maybe we can make this more > usable by improving the error message to show which py.test command to run > to repeat the individual test unit. Otherwise we should break them up into > separate files. pytest now writes junit files itself. Therefor these tests are all split up in Jenkins and one can only select the failing test cases and get a nice test output. ucs-test (10.0.6-1) 9c88f01a5949 | Bug #40709: let pytest write junit files instead of ucs-test commit 9c88f01a594937a379ffc3f56ed050cf2b00c496 (HEAD -> 5.0-0, origin/5.0-0) Author: Florian Best <best@univention.de> Date: Fri May 3 21:25:13 2019 +0200 Bug #40709: let pytest write junit files instead of ucs-test ucs-test writes junit files per test file. py-test separates each test into a own junit test section. to improve readability and debugability we don't let ucs-test writes junit files for pytests and force pytest to create them. Alternatively we could write the file to a different location, to also include the old variant. In case pytest failed to write the junit file, ucs-test creates one instead (to make sure we detect errors in the pytest definition). Missing is integration for /usr/share/ucs-test/runner but pytest can be invoked directly instead of calling the test directly. Missing is also the suffix "$hostname" in the test name. I think it's not really needed. The Jenkins URL already includes them. Old commits: ucs-test (7.0.20-3) r77885 | Bug #43862: Bug #40709: allow .py suffix for pytest-tests r77884 | Bug #43862: Bug #40709: fixup boolean logic r77882 | Bug #43862: Bug #40709: allow .py suffix for pytest-tests
The integration has been enhanced further: * use @pytest.mark.tags('apptest') to add single tags for tests * use @pytest.mark.exposure('dangerous') to mark single tests as safe/careful/dangerous * use @pytest.mark.roles('domaincontroller_master', 'domaincontroller_backup') and/or @pytest.mark.roles_not('memberserver') to exclude certain roles for some tests * no need to write the full path /usr/bin/pytest-3 into the test header "pytest-3" is enough Introduced was a global univention.testing.conftest which is a pytest plugin providing the above features. This makes it necessary that if one uses a custom conftest.py one needs to add to it the following line: pytest_plugins = ["univention.testing.conftest"] Next steps: * provide @pytest.mark.skip_package('univention-s4-connector', 'univention-samba4') * provide @pytest.mark.require_package('univention-s4-connector', 'univention-samba4') * add commonly used fixtures like "ucr" "udm", etc. to global conftest.py ucs-test (10.0.6-9) dbbd1864dfd8 | fixup! Bug #40709: implement pytest marker for ucs-test exposure,roles,tags 3bfedd023863 | Bug #40709: implement pytest marker for ucs-test exposure,roles,tags b608b3c47d64 | Bug #40709: allow ucs-test checks to extend pytest arguments ba0ac2d48ce4 | Bug #40709: detect path of pytests automatically
Maybe we should register the conftest.py via the python entry point mechanism (https://packaging.python.org/specifications/entry-points/).
I have deactived the feature in UCS 4.4-8, as it crashes all tests there. Pytest v3 does not have _pytest.main.Node.iter_markers, which leads to: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/univention/testing/conftest.py", line 88, in pytest_runtest_setup check_tags(item) File "/usr/lib/python2.7/dist-packages/univention/testing/conftest.py", line 96, in check_tags tags = set(tag for mark in item.iter_markers(name="tags") for tag in mark.args) or set(item.config.getoption("--ucs-test-default-tags", [])) AttributeError: 'Function' object has no attribute 'iter_markers' [4.4-8 0e1319c4e7] Bug #40709: temporarily deactivate feature as Pytest v3 does not have _pytest.main.Node.iter_markers ucs-test (9.0.7-65)
The currently symlinked/copied conftest.py has been moved to /usr/share/ucs-test/ so that all tests can use them. 4.4-8: ucs-test (9.0.7-68) de1f2a94d6fb | Bug #40709: [pytest] share one common conftest.py 5.0-0: ucs-test (10.0.6-51) 1245d0413e7c | Bug #40709: [pytest] share one common conftest.py
[5.0-0] 5930bf1021 refactor[ucs-test] Drop several fixtures test/ucs-test/tests/60_umc/102_test_umc_security.py | 9 ++++++--- test/ucs-test/tests/60_umc/103_test_umc_server_down.py | 11 +++++++---- test/ucs-test/tests/60_umc/104_expired_password.py | 19 +++++++++++-------- test/ucs-test/tests/60_umc/105_change_expired_password_fail_reason.py | 11 +++++++---- test/ucs-test/tests/conftest.py | 20 -------------------- 5 files changed, 31 insertions(+), 39 deletions(-) [5.0-0] 6ef6e9137f doc[ucs-test] Use of pytest test/ucs-test/README.md | 15 +++++++++- test/ucs-test/tests/conftest.py | 75 +++++++++++++++++++++++++++++++++++++++++++++++-- test/ucs-test/univention/testing/strings.py | 2 +- test/ucs-test/univention/testing/utils.py | 28 ++++++++++-------- 4 files changed, 103 insertions(+), 17 deletions(-)
OK