Univention Bugzilla – Bug 50500
Diagnostic modules for UCS@school
Last modified: 2020-03-25 16:09:25 CET
We need some diagnostic modules for UCS@school
Created first WIP version of ucs-school-umc-diagnostic. → I think it needs a build dependency to a newer version of univention-management-console-dev → translations are missing → wording should be improved → test scripts should be written
I like the addition of u@s diagnostic modules. But I realized that there is at least some shared functionality between the fix functions of the plugins and several scripts in u@s that do similar work (in particular the computer role fix). I would request to refactor that shared code into a common module/package in ucsschool to prevent code duplication.
(In reply to Ole Schwiegert from comment #2) > I like the addition of u@s diagnostic modules. But I realized that there is > at least some shared functionality between the fix functions of the plugins > and several scripts in u@s that do similar work (in particular the computer > role fix). > > I would request to refactor that shared code into a common module/package in > ucsschool to prevent code duplication. We had already discussed this last year, but unfortunately we did not document the result here at the bug. As far as I remember: our lib methods need information for correct functioning, which is explicitly not available here. 900_ucsschool_slave_groupmemberships ==================================== The module checks if the domaincontroller slave and memberserver objects are members of both groups: DC-Edukativnetz + OU-$OU-DC-Edukativnetz or DC-Verwaltungsnetz + OU-$OU-DC-Verwaltungsnetz or Member-Edukativnetz + OU-$OU-Member-Edukativnetz or Member-Verwaltungsnetz + OU-$OU-Member-Verwaltungsnetz Inconsistencies in these group memberships can trigger erratic behaviour of UCS@school. The following inconsistency checks are currently implemented: 1) Host object is member in global %s group but not in OU specific %s group (or the other way around) 2) Host object is member in edu groups AND in admin groups which is not allowed 3) Slave object is member in memberserver groups 4) Memberserver object is member in slave groups There is *no* button implemented, to fix existing errors (IMHO this can only be done manually). WARNING: is is not (yet) checked if the group memberships match to the ucsschoolRole attribute! 901_ucsschool_import_counter ============================ This module checks the counter objects of the UCS@school import. Counter objects may exist for usernames (cn=unique-usernames) or mail addresses (cn=unique-email). (Unfortunately the LDAP attribute is called "ucsschoolUsernameNextNumber" but is used for usernames and mail addresses. There are some constraints for the attribute ucsschoolUsernameNextNumber that are checked by the module: 1) ucsschoolUsernameNextNumber is a integer 2) ucsschoolUsernameNextNumber is 2 or higher 3) ucsschoolUsernameNextNumber is higher than highest suffix number of user with same (username/mailPrimaryAddress) prefix Especially the last check makes sure that there are no import errors in the future due to conflicting usernames/mail addresses. There is *no* button implemented, to fix existing errors (IMHO this can only be done manually). 902_ucsschool_ou_consistency ============================ This module checks the UCS@school OU objects: - only OUs with correct objectclass ("ucsschoolOrganizationalUnit") are checked - is ucsschoolRole set? - is ucsschoolRole set to "school:school:$OU"? - is a displayName set? - is a HomeShareFileServer and a ClassShareFileServer set? - is a HomeShareFileServer and a ClassShareFileServer (not) set to the DC master in multi/single server environment? - do HomeShareFileServer and ClassShareFileServer contain valid/existing DNs? There is *no* button implemented, to fix existing errors.
903_ucsschool_schoolcomputers ============================= This module checks the computer objects within each school: if the object class "ucsschoolComputer" or the attribute "ucsschoolRole" is missing, a critical error is shown. Via a "fix" button, the faulty object can be corrected.
QA diagnose modules 900-903 Summary: After each value change I made, I ran the diagnose modul in question, noted the errors,change the values back to their original value. Since no errors were thrown after changing the values back, I skipped this information for simplicity. To change the values I used ldapvi and ldapmodify. Remarks: 902_ucsschool_ou_consistency.py catch exception instead of retrieving empty list 903_ucsschool_schoolcomputers.py computers/mac -> computers/macos ========================================= ## 900_ucsschool_slave_groupmemberships.py I removed a slave-dc from group dc-edukativnetz and got the following: ---------------------------------------- The following objects have faulty group memberships: cn=ucs-4064,cn=dc,cn=server,cn=computers,ou=testschool1,dc=wenzel-univention,dc=intranet - Host object is member in global edu group but not in OU specific slave group (or the other way around) ---------------------------------------- Added member-edukativ to OUtestschool1-Member-Verwaltungsnetz and (afterwards) DC-Verwaltungsnetz. Both correctly raised this error: ---------------------------------------- The following objects have faulty group memberships: - Host object is member in edu groups AND in admin groups which is not allowed ---------------------------------------- I added the slave to the memberserver group: ---------------------------------------- The following objects have faulty group memberships: cn=ucs-4064,cn=dc,cn=server,cn=computers,ou=testschool1,dc=wenzel-univention,dc=intranet - Host object is member in global edu group but not in OU specific memberserver group (or the other way around) - Slave object is member in memberserver groups ---------------------------------------- I added the Memberserver to the slave-group: ---------------------------------------- univention.management.console.modules.diagnostic.Warning: UCS@school Domaincontroller Slave objects rely on the membership within certain UCS@school LDAP groups. Inconsistencies in these group memberships can trigger erratic behaviour of UCS@school. The following objects have faulty group memberships: cn=ucs-8036,cn=computers,dc=wenzel-univention,dc=intranet - Host object is member in global edu group but not in OU specific slave group (or the other way around) - Memberserver object is member in slave groups ---------------------------------------- # 901_usschool_import_counter.py 1) **ucsschoolUsernameNextNumber is a integer** -> I could not change the value to something unlike an integer. 2) **ucsschoolUsernameNextNumber is 2 or higher:** I set the counter to 1 (same with 0): ---------------------------------------- cn=juhani.bandermann,cn=unique-usernames,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet - cn=juhani.bandermann,cn=unique-usernames,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet: counter=1 ---------------------------------------- 3)** ucsschoolUsernameNextNumber is higher than highest suffix number of user with same username prefix** I created a user with username dave.gehl3 in the UMC and ran the diagnose modul: ---------------------------------------- univention.management.console.modules.diagnostic.Warning: UCS@school stores internal counters for the next free username or mail address. Inconsistencies in these counters can trigger erratic behaviour of the UCS@school import. The following objects have faulty counter values: cn=dave.gehl,cn=unique-usernames,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet - cn=dave.gehl,cn=unique-usernames,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet: usernames counter=2 but found user with uid dave.gehl3 ---------------------------------------- 1) **ucsschoolUsernameNextNumber is a integer** -> I could not change the value to something unlike an integer. 2) **ucsschoolUsernameNextNumber is 2 or higher** I set the ucsschoolUsernameNextNumber value to 1 (same with 0): ---------------------------------------- The following objects have faulty counter values: cn=ema.pohl,cn=unique-email,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet - cn=ema.pohl,cn=unique-email,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet: counter=1 ---------------------------------------- 3) **ucsschoolUsernameNextNumber is higher than highest suffix number of user with same (username/mailPrimaryAddress) prefix** ---------------------------------------- The following objects have faulty counter values: cn=rutmar.kloep,cn=unique-email,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet - cn=rutmar.kloep,cn=unique-email,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet: email counter=2 but found user with uid rutmar.kloep3 ---------------------------------------- ## 902_ucsschool_ou_consistency.py **is ucsschoolRole set?** I removed the attribute uccschoolRole and got the following: ---------------------------------------- - ucsschoolRole is not set - ucsschoolRole "school:school:testschool3" not found ---------------------------------------- **is ucsschoolRole set to "school:school:$OU"?** I changed school to the non-existent testschool5: ---------------------------------------- - ucsschoolRole "school:school:testschool3" not found ---------------------------------------- **is a displayName set?** I removed the attribute displayname. The following problems were found: ---------------------------------------- - displayName is not set ---------------------------------------- **is a HomeShareFileServer and a ClassShareFileServer set?** I removed the attributes homesharefileserver and ucsschoolClassShareFileServer. The following problems were found: ---------------------------------------- - ucsschoolHomeShareFileServer is not set - ucsschoolClassShareFileServer is not set ---------------------------------------- **is a HomeShareFileServer and a ClassShareFileServer (not) set to the DC master in multi/single server environment?** I changed the values of HomeShareFileServer & ClassShareFileServer (in a multi-server environment). ---------------------------------------- - ucsschoolHomeShareFileServer is set to master in a UCS@school multi server environment - ucsschoolClassShareFileServer is set to master in a UCS@school multi server environment ---------------------------------------- I set the value ucsschoolHomeShareFileServer to another existing client computer. ucsschoolHomeShareFileServer: cn=macdoof,cn=computers,ou=DEMOSCHOOL,dc=wenzel-univention,dc=intranet The following problems were found: ---------------------------------------- - ucsschoolHomeShareFileServer is not set to master in a UCS@school single server environment ---------------------------------------- I set the value of ucsschoolClassShareFileServer to another existing client computer. ucsschoolClassShareFileServer: cn=macdoof,cn=computers,ou=DEMOSCHOOL,dc=wenzel-univention,dc=intranet The following problems were found: ---------------------------------------- - ucsschoolClassShareFileServer is not set to master in a UCS@school single server environment ---------------------------------------- I set the value to an object, which is not present in ldap. The code if not lo.search(base=value, scope='base'): Does not return an empty list but throws an error. Suggestion: try: invalid_ldap_value = not lo.search(base=value, scope='base') except ldap.NO_SUCH_OBJECT: invalid_ldap_value = True if invalid_ldap_value: After changing the code to the suggested solution, i got the expected result (...intrane instead of intranet): ---------------------------------------- - ucsschoolClassShareFileServer contains invalid value: 'cn=dctestschool-01,cn=dc,cn=server,cn=computers,ou=testschool,dc=wenzel-univention,dc=intrane' ---------------------------------------- ## 903_ucsschool_schoolcomputers.py I removed ucsschoolComputer and/or the ucsschoolRole. This was done for windows, macos, ipmanagedclient, linux and ubuntu. I got the following output (or similar) for windows, ipmanagedclient, linux and ubuntu. ---------------------------------------- The following host objects do not contain all the necessary LDAP attributes for UCS@school: cn=win03,cn=computers,ou=testschool2,dc=wenzel-univention,dc=intranet ---------------------------------------- Please change computers/mac -> computers/macos to make it work for macos, too. **Fixing** After I removed ucsschoolComputer and/or the ucsschoolRole, then I ran fix_computers(None) This was done for windows, macos, ipmanagedclient, linux and ubuntu. ---------------------------------------- 12.03.20 02:22:35.438 MODULE ( PROCESS ) : Fixing cn=win03,cn=computers,ou=testschool2,dc=wenzel-univention,dc=intranet 12.03.20 02:22:35.439 MODULE ( PROCESS ) : old_val = ['krb5KDCEntry', 'krb5Principal', 'person', 'posixAccount', 'sambaSamAccount', 'shadowAccount', 'top', 'univentionHost', 'univentionObject', 'univentionWindows'] 12.03.20 02:22:35.439 MODULE ( INFO ) : old_val = None 12.03.20 02:22:35.439 MODULE ( INFO ) : role = 'win_computer:school:testschool2' 12.03.20 02:22:35.439 MODULE ( INFO ) : ml=[['objectClass', ['krb5KDCEntry', 'krb5Principal', 'person', 'posixAccount', 'sambaSamAccount', 'shadowAccount', 'top', 'univentionHost', 'univentionObject', 'univentionWindows'], ['krb5KDCEntry', 'krb5Principal', 'person', 'posixAccount', ---------------------------------------- Afterwards, the diagnose module didn't raise exceptions and the attribute could be found in the UMC and via univention-ldapsearch like ---------------------------------------- univention-ldapsearch cn=win03 objectClass: ucsschoolComputer ucsschoolRole: win_computer:school:testschool2 ----------------------------------------
> # 901_usschool_import_counter.py > > > 1) **ucsschoolUsernameNextNumber is a integer** > -> I could not change the value to something unlike an integer. > I could reproduce the case by removing the attribute completely. The following objects have faulty counter values: cn=y.wp1400rgej,cn=unique-usernames,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet - cn=y.wp1400rgej,cn=unique-usernames,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet: counter= cn=h.s4t1nvm29z,cn=unique-email,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet cn=z.u9q16ps7zm,cn=unique-email,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet - cn=z.u9q16ps7zm,cn=unique-email,cn=ucsschool,cn=univention,dc=wenzel-univention,dc=intranet: counter=
902_ucsschool_ou_consistency.py ============================= (Line 72) Add `if ucsschoolRoles` to catch cases without ucsschoolRoles in school A as well as school:school:non-existent in school B without getting an error. if ucsschoolRoles and not any(x.startswith('school:school:{}'.format(ou_attrs.get('ou')[0])) for x in ucsschoolRoles): (Line 81) Similar issue: add `continue`. If ucsschoolHomeShareFileServer or ucsschoolClassShareFileServer are not set, you will get 2 errors, since the value is 'not set' and 'not set to master (...) in a single server environment'. for attr_name in ('ucsschoolHomeShareFileServer', 'ucsschoolClassShareFileServer'): value = ou_attrs.get(attr_name, [None])[0] if not value: problematic_objects.setdefault(ou_dn, []).append(_('{0} is not set').format(attr_name)) continue
[twenzel/50500_diagnosemodules] 01fe00a97 Bug #50500: ucs-test for 901_ucsschool_ou_consistency ucs-test-ucsschool/90_ucsschool/260_ucsschool_ou_consistency =========================== ucs-test to check diagnose-module 902_ucsschool_ou_consistency.py. Idea: Set ldap-values, such that the diagnose-module raises warnings and check if the correct warnings were raised. The class UCSSchoolOuConsistencyCheck is derived from AutoMultiSchoolEnv, which contains multiple schools (A-C) as well as client computers. After changing a value, the expected Warning is added to the array of expected_errors. The changes made in case of running the script in a single server environment were added for the sake of completeness, since the test will be usually run in a multi server environment. ----------------------------- Ou A: - remove ucsschoolRole - remove displayName - remove ucsschoolHomeShareFileServer - remove ucsschoolClassShareFileServer Ou B: - set ucsschoolRole to school:school:non-existent (In single server environment) - set ucsschoolHomeShareFileServer to self.schoolB.winclient.dn - set ucsschoolHomeShareFileServer to self.schoolB.winclient.dn Ou C: (In multi server environment) - set ucsschoolHomeShareFileServer to ldap/hostdn - set ucsschoolHomeShareFileServer to ldap/hostdn ----------------------------- After calling 902_ucsschool_ou_consistency.py, an exception is raised if any of the expected warnings was not raised.
[twenzel/50500_diagnosemodules] 7cd341ed9 Bug #50500: Search for more specific warnings in import_counter ucs-test. [twenzel/50500_diagnosemodules] eb008d779 Bug #50500: ucs-tests: Also raise error if no warnings were raised by diagnose modules. [twenzel/50500_diagnosemodules] 01fe00a97 Bug #50500: ucs-test for 901_ucsschool_ou_consistency [twenzel/50500_diagnosemodules] 8bc79f393 Bug #50500: Add ucs-test for import counter ucs-test-ucsschool/90_ucsschool/143_import_counter =========================== ucs-test to check diagnose-module 901_ucsschool_import_counter.py. Set ldap-values, such that the diagnose-module raises warnings and check if the correct warnings were raised. The diagnose-module searches for all users with an ucsschoolRole: '(&(univentionObjectType=users/user)(ucsschoolRole=*))' so it suffices to only check against one (e.g. six students, which are used in this case). The class Test is derived from UniqueObjectTester, which has one school. After changing a value, the expected Warning is added to the array of expected_errors. ----------------------------- Each of the changes is done for one person: - change mailPrimaryAddress to mail3@domain - change uid to username3 - change ucsschoolUsernameNextNumber of unique-mail object to 0 - remove ucsschoolUsernameNextNumber of unique-mail object - change ucsschoolUsernameNextNumber of unique-usernames object to 0 - remove ucsschoolUsernameNextNumber of unique-usernames object ----------------------------- After calling 901_ucsschool_import_counter.py, an exception is raised if any of the expected warnings was not raised or none at all. Remark: The unique-objects are not removed afterwards. If the test runs in a closed environment, this should not be a problem. There is a small chance of running into problems if a user with the same random name exists and is tested against something similar.
(In reply to Tobias Wenzel from comment #5) > Remarks: > 902_ucsschool_ou_consistency.py > catch exception instead of retrieving empty list Fixed. > 903_ucsschool_schoolcomputers.py > computers/mac -> computers/macos Fixed. > 1) **ucsschoolUsernameNextNumber is a integer** > -> I could not change the value to something unlike an integer. You are right. The LDAP scheme only allows integer values. But I left the check as it is. I fixed the package ucs-school-umc-diagnostic and removed unused build settings/dependencies. Package: ucs-school-umc-diagnostic Version: 1.0.0-7A~4.4.0.202003191800 Branch: ucs_4.4-0 Scope: ucs-school-4.4 [4.4] 1256b262a Bug #50500: add advisory [4.4] f220b93fb Bug #50500: bump version [4.4] 5821a5d23 Bug #50500: remove unused build settings/dependencies [4.4] 3d2348a26 Bug #50500: add missing .install file [4.4] 5dd033975 Bug #50500: bump version [4.4] e1d753caf Bug #50500: cleanup [4.4] dee9e711a Bug #50500: add version specific dependency [4.4] 7e1815ffd Bug #50500: fixed wrong UDM module name [4.4] e4e20ece2 Bug #50500: catch exception if object is missing [4.4] 85862bcee Bug #50500: some cleanup / more specific error messages [4.4] 6d40ec2c7 Bug #50500: add absolute_import
[twenzel/50500_diagnosemodules] 5631d0dc1 Bug #50500: ucs-test for 903_ucsschoolcomputers diagnose ucs-test-ucsschool/90_ucsschool/403_ucsschool_schoolcomputers =========================== ucs-test to check diagnose-module 903_ucsschool_computers.py. Set ldap-values, such that the diagnose-module raises warnings and check if the correct warnings were raised. The class UCSSchoolSchoolComputer is derived from AutoMultiSchoolEnv, which contains multiple schools (A-C) as well as windows-client computers. To test all cases, I added two computers of macos, ipmanagedclient, linux and ubuntu. After changing a value, the computer-dn is added to the array of strings, which should be present in the diagnose-module warnings. ----------------------------- To test all cases the following two modifications are made for each computer-type: - remove ucsschoolRole - remove ucsschoolComputer of the objectClass (and ucsschoolRole, otherwise this wouldn't be possible) ----------------------------- After calling 903_ucsschool_computers.py, an exception is raised if any of the expected dns was inside the warnings or none at all.
ackage: ucs-school-umc-diagnostic Version: 1.0.0-9A~4.4.0.202003241232 Branch: ucs_4.4-0 Scope: ucs-school-4.4 [4.4] 0e448ee8b Bug #50500: fix logic error in diagnostic/plugins/902_ucsschool_ou_consistency.py [4.4] 40340e9fe Bug #50500: update advisory
Merged branch twenzel/50500_diagnosemodules into 4.4 [4.4] ebe824f0c Bug #50500: changelog [4.4] a4eb95691 Bug #50500: Add changelog entry [4.4] 341c9ed6f Bug #50500: Merge branch 'twenzel/50500_diagnosemodules' into 4.4 [4.4] 3ef194103 Bug #50500: Rename ucs-tests [4.4] 8bb1969d0 Bug #50500: ucsschool diagnosemodule 900 [4.4] de7538f2b Bug #50500: ucs-test: call diagnosic test by diagnostic.Instance [4.4] 65f785a57 Bug #50500: Change diagnose-module to diagnostic tool [4.4] 5d2ab526d Bug #50500: ucs-test for 903_ucsschoomcomputers fix [4.4] 074231feb Bug #50500: ucs-test for 903_ucsschoolcomputers diagnose [4.4] 385831d92 Bug #50500: Search for more specific warnings in import_counter ucs-test. [4.4] 6d16b36a9 Bug #50500: ucs-tests: Also raise error if no warnings were raised by diagnose modules. [4.4] 54dea3419 Bug #50500: ucs-test for 901_ucsschool_ou_consistency [4.4] dafd6699f Bug #50500: Add ucs-test for import counter [4.4] 81edc0cff Bug #50500: QA Suggestions [4.4] 1256b262a Bug #50500: add advisory [4.4] f220b93fb Bug #50500: bump version [4.4] 5821a5d23 Bug #50500: remove unused build settings/dependencies [4.4] 3d2348a26 Bug #50500: add missing .install file [4.4] 5dd033975 Bug #50500: bump version [4.4] e1d753caf Bug #50500: cleanup [4.4] dee9e711a Bug #50500: add version specific dependency [4.4] 7e1815ffd Bug #50500: fixed wrong UDM module name [4.4] e4e20ece2 Bug #50500: catch exception if object is missing [4.4] 85862bcee Bug #50500: some cleanup / more specific error messages [4.4] 6d40ec2c7 Bug #50500: add absolute_import [4.4] 2f9025285 Bug #50500: rewrite of import counter checks (speed + logic errors) [4.4] 6ac4b2953 Bug #50500: make existing scripts executable [4.4] a7284acd6 Bug #50500: loosen build dependency for now [4.4] e8e8968af Bug #50500: loosen build dependency for now [4.4] e82fd0d5c Bug #50500: add WIP version of package ucs-school-umc-diagnostic Package: ucs-test-ucsschool Version: 6.0.106A~4.4.0.202003241214 Branch: ucs_4.4-0 Scope: ucs-school-4.4 User: twenzel
Added "Recommends" entry to ucs-school-master and ucs-school-singlemaster, so ucs-school-umc-diagnostic is automatically installed on DC Master/DC Backup systems. Package: ucs-school-metapackage Version: 12.0.3-1A~4.4.0.202003251252 Branch: ucs_4.4-0 Scope: ucs-school-4.4 [4.4] e2a380f41 Bug #51004, #50500: update changelog [4.4] 3bf416dee Bug #50500: update advisory [4.4] c7694df6e Bug #51004, #50500: update changelog due to updated metapackage [4.4] 645bdc31d Bug #50500: update advisory [4.4] af46cdb3e Bug #50500: add Recommends to ucs-school-(single)master for ucs-school-umc-diagnostic
Installes and updates fine.
UCS@school 4.4 v5 has been released. https://docs.software-univention.de/changelog-ucsschool-4.4v5-de.html If this error occurs again, please clone this bug.