Univention Bugzilla – Bug 49415
stabilize errata tests on S4 connector
Last modified: 2023-01-13 11:47:01 CET
I added a generic wait_for() function, which takes all necessary parameters. 585c7514bbb0 | Bug #49415: add wait_for(replication, replication_postrun, drs_replication, s4_connector, verbose) 879ae81aa357 | Bug #49415: use wait_for_s4connector() instead of sleep(17) in wait_for_connector_replication() 98f540e6a4f5 | Bug #49415: use upstream wait function
Created attachment 10006 [details] draft_wait_for.patch Thanks for doing the first steps in this issue. I'd like to go a step further and have * the fronted functions make decisions about what to wait for based on server/role and installed services * the backend functions iterating through a list of things to wait for * supporting waiting for both (or more) directions of replication (in the right order) The attached untested draft would define two frontend functions: * wait_for_replication_from_master_openldap_to_local_samba * wait_for_replication_from_local_samba_to_local_openldap That would probably cover most use cases.
(In reply to Arvid Requate from comment #2) > Created attachment 10006 [details] > draft_wait_for.patch > > Thanks for doing the first steps in this issue. I'd like to go a step > further and have > > * the fronted functions make decisions about what to wait for > based on server/role and installed services > > * the backend functions iterating through a list of things to wait for > > * supporting waiting for both (or more) directions of replication (in the > right order) > > > The attached untested draft would define two frontend functions: > > * wait_for_replication_from_master_openldap_to_local_samba > * wait_for_replication_from_local_samba_to_local_openldap > > That would probably cover most use cases. I applied your patch with some additional enhancements. Not sure if I like this utils.wait_for([(utils.LISTENER, 'replication')]) enum style, but we'll see. In theory you can now pass conditions=[DRS, LISTENER] which is the wrong order. I think DRS depends on LISTENER. Can we use your new functions for udm.{create,modify,remove}_object()?
The test case 50_dns_reverse_zone_check_soa_record_serial_incrementation was adapted, so that it explicit waits for s4 connector and then verifies the s4-modified-replicated attributes. This currently still fails. So wait_for_s4_connector() is not really working. I will test what happens if I add wait_for_drs_replication(filter=...). diff --git a/test/ucs-test/tests/67_udm-dns/50_dns_reverse_zone_check_soa_record_serial_incrementation b/test/ucs-test/tests/67_udm-dns/50_dns_reverse_zone_check_soa_record_serial_incrementation index e27d287d57..f8de71e196 100755 --- a/test/ucs-test/tests/67_udm-dns/50_dns_reverse_zone_check_soa_record_serial_incrementation +++ b/test/ucs-test/tests/67_udm-dns/50_dns_reverse_zone_check_soa_record_serial_incrementation @@ -10,10 +10,13 @@ ## 3.1-1: skip ## 3.2-0: fixed +import ldap.dn +import ldap.filter + import univention.testing.utils as utils import univention.testing.udm as udm_test import univention.testing.strings as uts -from univention.testing.ucs_samba import wait_for_s4connector +from univention.testing.ucs_samba import wait_for_replication_from_master_openldap_to_local_samba if __name__ == '__main__': with udm_test.UCSTestUDM() as udm: @@ -33,7 +36,7 @@ if __name__ == '__main__': reverse_zone_properties['ttl'] = '12' udm.modify_object('dns/reverse_zone', dn=reverse_zone, ttl=reverse_zone_properties['ttl']) - wait_for_s4connector() + wait_for_replication_from_master_openldap_to_local_samba(ldap_filter=ldap.filter.filter_format('DC=%s', [ldap.dn.str2dn(reverse_zone)[0][0][1]])) utils.verify_ldap_object(reverse_zone, {'sOARecord': ['%s %s. %s %s %s %s %s' % ( reverse_zone_properties['nameserver'] + '.' if utils.package_installed('univention-s4-connector') else reverse_zone_properties['nameserver'], reverse_zone_properties['contact'].replace('@', '.'), @@ -55,7 +58,7 @@ if __name__ == '__main__': reverse_zone_properties['ttl'] = '12' udm.modify_object('dns/reverse_zone', dn=reverse_zone, ttl=reverse_zone_properties['ttl']) - wait_for_s4connector() + wait_for_replication_from_master_openldap_to_local_samba(ldap_filter=ldap.filter.filter_format('DC=%s', [ldap.dn.str2dn(reverse_zone)[0][0][1]])) utils.verify_ldap_object(reverse_zone, {'sOARecord': ['%s %s. %s %s %s %s %s' % ( reverse_zone_properties['nameserver'] + '.' if utils.package_installed('univention-s4-connector') else reverse_zone_properties['nameserver'], reverse_zone_properties['contact'].replace('@', '.'), I hope this will fix the issues. If yes, I think we can integrate this function call more upstream in udm.py:modify(): There a mapping of the RDN-value between S4<->LDAP should be further adapted. DNS objects have {zoneName,relativeDomainName,..} = DC. User objects have uid = CN. etc. Unfortionately I cannot even reproduce this error on my machine when running this test since hours in a loop. Any ideas how I could force it to fail more often? Less RAM? More LDAP noise?
Created attachment 10019 [details] always_wait_for_s4_connector_on_master.patch The attached proposal would make the UCSTestUDM methods always wait for the S4-Connector. I'm not sure if this is the right approach, as it will add a lot of useless waits. I mean, in each tests case we create a couple of objects and it would be great not to wait between each udm operation, but just once after the test setup is finished and before the actual test starts. During the actual test, the automatic "wait for S4-Connector by default" might make sense. To speed things up, I proposed Bug #49442. When we have implemented something like that, we can reduce the s4cooldown_t to 1 iteration in wait_for_s4connector. That would be feasable I guess. In the attached patch I also added a sketch for a UCSTestUDM.wait_for_ng function that takes server/role into consideration. We should discuss ideas how to improve this. In an ideal (phantasy) world I would imaging that all udm operations return some operation GUID to me, which is then propagated through LDAP and S4-Connector replication. At least in non-conurrent test-case situations that would allow me to check if the operation has arrived at the destination. At least in most cases (counter example: S4-Connector ping-pong of the objectSid during user creation).
Regarding Comment 4 / wait_for_drs_replication: It's pretty much impossible to make this generic. The current approach is to craft an LDAP filter for Samba/AD that matches the object state after successful DRS replication. That's a pretty specific thing for each test case. For object creation it's pretty simple: guess the right RDN and hope that it's unique and wait for it to appear on the target system. But for modifications, uh-oh ..., each test case has to tell wait_for_drs_replication what exact value to wait for. And that's an AD-style encoded value. For DNS objects, you pretty much need the stuff from univention-s4-connector/modules/univention/s4connector/s4/dns.py to encode the values. IMHO that pretty much kills this approach for dns-objects. For them it's much simpler and much more significant if the DNS-server can finally resolve to the correct value.
c65709cd278bed36ce0ddca019cf9537909ecfb6 - 61_udm-users/100_test_users.py::test_modlist_samba_bad_pw_count
Note: A lot of commits were done with the wrong bug number #49414!
ef7ac61b29e3053bebf12b237dc97681ed62406f - ucs-test wait_for_s4connector_replication in 72_udm-extensions/40_test_udm_hook
Some test cases now use wait_for: * Bug #49415: Use wait_for in some s4 connector corner cases
Another case: [4.4-0 3012b08513] * Bug #49415: Use wait_for in 45udm-cli
Another case: [4.4-0 68b04919e7] * Bug #49415: Use udm test class and wait_for in 01_base/07kpasswd_output
*** Bug 47345 has been marked as a duplicate of this bug. ***
*** Bug 48674 has been marked as a duplicate of this bug. ***
*** BEGIN *** ['/usr/bin/pytest-3', '02_test_dns_resolve.py', '--continue-on-collection-errors', '--tb=native', '--color=auto', '--confcutdir=/usr/share/ucs-test/', '--ucs-test-default-tags', 'skip_admember', '--ucs-test-exposure', 'dangerous', '--ucs-test-default-expos *** 67_udm-dns/02_test_dns_resolve.py *** Check DNS resolving of all DNS record and zone types *** *** START TIME: 2021-09-30 16:36:32 *** ============================= test session starts ============================== platform linux -- Python 3.7.3, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 rootdir: /usr/share/ucs-test/67_udm-dns, inifile: plugins: cov-2.6.0 collected 8 items 02_test_dns_resolve.py .......F [100%] =================================== FAILURES =================================== ______________ Test_DNSResolve.test__dns_ns_record_check_resolve _______________ Traceback (most recent call last): File "/usr/share/ucs-test/67_udm-dns/02_test_dns_resolve.py", line 360, in test__dns_ns_record_check_resolve assert nameservers == found, "Record not found: %s" % ([set(nameservers) - set(found)],) AssertionError: Record not found: [{'y9slaqdl7r.phahn.qa', 'vx4lew82kt.phahn.qa'}] assert ['y9slaqdl7r....2kt.phahn.qa'] == [] Left contains more items, first extra item: 'y9slaqdl7r.phahn.qa' Use -v to get the full diff ----------------------------- Captured stdout call ----------------------------- Creating dns/ns_record object with /usr/sbin/udm-test dns/ns_record create --superordinate zoneName=phahn.qa,cn=dns,dc=phahn,dc=qa --set zone=tecmjnok5c --set zonettl=378 --append nameserver=y9slaqdl7r.phahn.qa --append nameserver=vx4lew82kt.phahn.qa wait_for_s4connector(): skip, univention-s4-connector not installed. --------------------------- Captured stdout teardown --------------------------- Performing UCSTestUDM cleanup... removing DN: relativeDomainName=tecmjnok5c,zoneName=phahn.qa,cn=dns,dc=phahn,dc=qa Cleanup: wait for replication and drs removal wait_for_drs_replication(): skip, univention-samba4 not installed. trying to restart UDM CLI server sending signal 15 to process 28152 (['/usr/bin/python3', '/usr/share/univention-directory-manager-tools/univention-cli-server']) process already terminated UCSTestUDM cleanup done ===================== 1 failed, 7 passed in 210.11 seconds ===================== *** END TIME: 2021-09-30 16:40:02 *** *** TEST DURATION (H:MM:SS.ms): 0:03:30.801609 *** *** END *** 1 ***
*** BEGIN *** ['/usr/bin/python', '07kpasswd_output'] *** *** 01_base/07kpasswd_output *** Check if kpasswd spuriously reports success This script tests if kpasswd spuriously reports success when trying to change the password to a too short one. It performs the following steps for the test: * Create user with "long password" * Log in with this user using ssh * Try to change password with kpasswd to "too short password" and parse its output * Test "long password" and "too short password" by trying to log in using ssh *** *** START TIME: 2021-09-30 15:17:09 *** Creating users/user object with /usr/sbin/udm-test users/user create --position cn=users,dc=phahn,dc=qa --set username=m0bnhwulom --set firstname=zl0zptsj7i --set lastname=ckk82c1x2b --set 'primaryGroup=cn=Domain Admins,cn=groups,dc=phahn,dc=qa' --set password=286d65b5e No shell prompt found! Output: '\r\nCreating directory \'/home/m0bnhwulom\'.\r\nUnivention Primary Directory Node 5.0-0:\r\n\r\nThe UCS management system is available at https://m30.phahn.qa/ (10.200.17.31)\r\n\r\nYou can log into the Univention Management Console - the principal tool to mana Cleanup after exception: <type 'exceptions.SystemExit'> 120 Performing UCSTestUDM cleanup... removing DN: uid=m0bnhwulom,cn=users,dc=phahn,dc=qa Cleanup: wait for replication and drs removal OpenLDAP object to check against S4-Connector match_filter doesn't exist: uid=m0bnhwulom,cn=users,dc=phahn,dc=qa DRS wait not required, S4-Connector match_filter did not match the OpenLDAP object: uid=m0bnhwulom,cn=users,dc=phahn,dc=qa trying to restart UDM CLI server sending signal 15 to process 12805 (['/usr/bin/python3', '/usr/share/univention-directory-manager-tools/univention-cli-server']) process already terminated UCSTestUDM cleanup done *** END TIME: 2021-09-30 15:17:23 *** *** TEST DURATION (H:MM:SS.ms): 0:00:14.132264 *** *** END *** 120 ***
https://jenkins2022.knut.univention.de/job/UCS-5.0/job/UCS-5.0-2/job/AutotestUpgrade/lastCompletedBuild/SambaVersion=s4,Systemrolle=slave/testReport/59_udm.72_test_udm_extensions/Test_UDMExtensions/test_test_py2_and_3_udm_hook/ > Creating users/user object with /usr/sbin/udm-test users/user create --position cn=users,dc=autotest075,dc=test --set username=a3dhxmeg6u --set lastname=unzbbi1z7u … > wait_for_s4connector(): skip, univention-s4-connector not installed. > . . . . . . Waiting for replication... > Done: replication complete. > Waiting for connector replication > wait_for_s4connector(): skip, univention-s4-connector not installed. > Modifying users/user object with /usr/sbin/udm-test users/user modify --dn uid=a3dhxmeg6u,cn=users,dc=autotest075,dc=test --set lastname=mmz054mlrx > Waiting for replication... > Done: replication complete. > Waiting for connector replication > wait_for_s4connector(): skip, univention-s4-connector not installed. > Modifying users/user object with /usr/sbin/udm-test users/user modify --dn uid=a3dhxmeg6u,cn=users,dc=autotest075,dc=test --set lastname=yf8c9jo2gm > Waiting for replication... > Done: replication complete. > Waiting for connector replication > wait_for_s4connector(): skip, univention-s4-connector not installed. > Exception occurred: <class 'univention.testing.utils.LDAPObjectValueMissing'> (DN: uid=a3dhxmeg6u,cn=users,dc=autotest075,dc=test sn: [b'mmz054mlrx'], missing: 'yf8c9jo2gm'; unexpected: 'mmz054mlrx' > description: [b'USERNAME=a3dhxmeg6u LASTNAME=mmz054mlrx'], missing: 'USERNAME=a3dhxmeg6u LASTNAME=yf8c9jo2gm'; unexpected: 'USERNAME=a3dhxmeg6u LASTNAME=mmz054mlrx' > ). Retrying in 10.00 seconds (retry 0/20). While S4C is not installed on the SLAVE, it still is on the MASTER and thus test/ucs-test/univention/testing/ucs_samba.py:175 def wait_for_s4connector() should wait for S4C@master + UDN/UDL replication. It looks like because of the multiple modifications via OpenLDAP@master to the same object S4C@master REVERTS some of those changes because it synchronizes the previous state from Samba4 back into OpenLDAP.