Univention Bugzilla – Bug 51955
Remove UVMM schema on upgrade
Last modified: 2023-09-25 17:50:15 CEST
For UCS-5 UVMM is removed, including the schema. This leads to problems during updates, as OpenLDAP then no longer knows anything about the objectClasses and attributes, which are still referenced by the LMDB: Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEPVINTERFACE" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILECPUS" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILENAMEPREFIX" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEVNC" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEPVDISK" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEPVCDROM" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILERAM" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEARCH" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEVIRTTECH" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEBOOTDEVICES" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEDISKSPACE" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEINTERFACE" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEKBLAYOUT" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILERTCOFFSET" inserted. Sep 08 09:27:34 m34 slapd[29182]: UNKNOWN attributeDescription "UNIVENTIONVIRTUALMACHINEPROFILEOS" inserted. We should check for it in preup.sh and link to <https://help.univention.com/t/remove-ldap-schema-extensions/6443>.
In univention-management-console-module-diagnostic/umc/python/diagnostic/plugins/60_old_schema_registration.py we register old schema files in the LDAP directory (ucs_registerLDAPExtension), so we just keep the old schema (no need to edit the database either) Maybe we should do something similar here?
I think we will support environments where the DC Master is already UCS 5 but UVMMd and KVM nodes can still run with UCS 4.4, so the schema cannot be removed.
(In reply to Erik Damrose from comment #2) > I think we will support environments where the DC Master is already UCS 5 > but UVMMd and KVM nodes can still run with UCS 4.4, so the schema cannot be > removed. Yes, it must not be removed automatically with 5.0. Maybe with 5.1.
(In reply to Ingo Steuwer from comment #3) > (In reply to Erik Damrose from comment #2) > > I think we will support environments where the DC Master is already UCS 5 > > but UVMMd and KVM nodes can still run with UCS 4.4, so the schema cannot be > > removed. > > Yes, it must not be removed automatically with 5.0. Maybe with 5.1. We're talking about still shipping the UVMM.schema extension with new UCS-5: Currently with UCS-4 that schema is ALWAYS included due to a hard-coded dependency on univention-uvmm-schema: management/univention-ldap/debian/control:13: univention-virtual-machine-manager-schema, virtualization/univention-virtual-machine-manager-daemon/debian/control:26: univention-virtual-machine-manager-schema (>= 5.0.3), virtualization/univention-virtual-machine-manager-node/debian/control:24: univention-virtual-machine-manager-schema, virtualization/univention-virtual-machine-manager-schema/debian/control:1:Source: univention-virtual-machine-manager-schema virtualization/univention-virtual-machine-manager-schema/debian/control:14:Package: univention-virtual-machine-manager-schema virtualization/univention-virtual-machine-manager-schema/debian/control:42: univention-virtual-machine-manager-schema, virtualization/univention-virtual-machine-manager-schema/debian/control:63: univention-virtual-machine-manager-schema, For NEW installations we definitely do NOT want to install it by default, so with UCS-5.0 that dependency and the package itself are REMOVED via Bug #51982 already. So this is only relevant for UPGRADES: There the solution proposed in comment 1 looks easiest to me, which then is a UCS-4.4-x task.
I added Python 3 compatibility for the UDM modules: univention-virtual-machine-manager-schema (9.0.2-11) 8ea6d22a5938 | Bug #51955: make UDM modules compatible with Python 3 univention-virtual-machine-manager-schema (9.0.2-10) 896af4a37f1b | Bug #51955: Register uvmm.schema in LDAP, remove conffiles
6daf2701 Readd several previously deleted files. This should fix issues with different package versions on multiple hosts and prevent occurrences of failed.ldif univention-virtual-machine-manager-schema 9.0.2-13A~4.4.0.202102182231 Waiting for jenkins results
Jenkins tests were successful. Overview of changes: We want to register the UVMM schema, ldap acls and udm modules+syntax in LDAP, because the UVMM packages are not built for UCS 5. But we need to support 4.x systems in the domain with UVMM still installed. Schema and ACLs: The locally included schema and schema registration with ucs_registerLDAPExtension has to happen in one step with only one slapd restart, otherwise a failed.ldif occurs on other domain servers. The UCR template for the schema can now be deactivated by UCR. This is done in the joinscript just prior to the registration of the schema file in LDAP. UDM: The package update removes the python files from /usr/lib/python2.7/dist-packages/univention/admin/..; while registering them via ucs_registerLDAPExtension replicates them to the exact same place. This works fine on the DC master, but for other server roles we need to once copy the modules to the correct place in the package postinst.
Update UCS 5 fail in jenkins the update itself worked, but after the update univention-run-join-scripts fails with EXITCODE=already_executed univention-join-hooks: looking for hook type "join/post-joinscripts" on master071.AutoTest071.local Traceback (most recent call last): File "/usr/share/univention-join/univention-join-hooks", line 171, in <module> main() File "/usr/share/univention-join/univention-join-hooks", line 113, in main udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 121, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 105, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/univention/admin/handlers/uvmm/info.py", line 88, in <module> syntax=udm_syntax.UvmmProfiles, AttributeError: module 'univention.admin.syntax' has no attribute 'UvmmProfiles'
I analyzed this (only a little bit): The order of registration seems correct. The missing syntax file is registered before the UDM modules. The "udm_extension.py" listener module might not preserve this order or fails with the syntax file but continues with the modules. It fails with a similar traceback because it is itself doing univention.admin.modules.update(). Before this happened it fails activating e.g. cn=66univention-ldap-server_acl-master-uvmm,cn=ldapacl,cn=univention,dc=autotest070,dc=local due to Bug #52815. Maybe this also causes the UDM syntax to fail. Let's see first if fixing Bug #52815 helps. In the logfile it seems the UDM syntax registration went well. 21.02.21 00:18:17.201 LISTENER ( PROCESS ) : updating 'cn=univention-virtual-machine-manager-schema,cn=udm_syntax,cn=univention,dc=autotest070,dc=local' command m 22007 21.02.21 00:18:32.226 LISTENER ( PROCESS ) : ldap_extension: Reloading LDAP server. -- https://jenkins.knut.univention.de:8181/job/UCS-5.0/job/UCS-5.0-0/view/Default/job/AutotestUpgrade/SambaVersion=no-samba,Systemrolle=master/ws/test/listener.log 21.02.21 00:49:45.960 LISTENER ( PROCESS ) : ldap_extension: cn=66univention-ldap-server_acl-master-uvmm,cn=ldapacl,cn=univention,dc=autotest070,dc=local active? [b'TRUE'] File "<stdin>", line 26 ''' % params ^ SyntaxError: invalid syntax File "<stdin>", line 26 ''' % params ^ SyntaxError: invalid syntax 21.02.21 00:49:57.146 LISTENER ( WARN ) : finished initializing module ldap_extension with rv=0 21.02.21 00:49:57.146 LISTENER ( WARN ) : initializing module udm_extension 21.02.21 00:49:59.549 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:01.119 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:03.388 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:05.059 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:06.457 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:07.730 LISTENER ( WARN ) : handler: udm_extension (failed) Traceback (most recent call last): File "/usr/lib/univention-directory-listener/system/udm_extension.py", line 203, in handler udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 121, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 105, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/univention/admin/handlers/uvmm/info.py", line 88, in <module> syntax=udm_syntax.UvmmProfiles, AttributeError: module 'univention.admin.syntax' has no attribute 'UvmmProfiles' Traceback (most recent call last): File "/usr/lib/univention-directory-listener/system/udm_extension.py", line 203, in handler udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 121, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 105, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/univention/admin/handlers/uvmm/info.py", line 88, in <module> syntax=udm_syntax.UvmmProfiles, AttributeError: module 'univention.admin.syntax' has no attribute 'UvmmProfiles' Traceback (most recent call last): File "/usr/lib/univention-directory-listener/system/udm_extension.py", line 203, in handler udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 121, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 105, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/univention/admin/handlers/uvmm/info.py", line 88, in <module> syntax=udm_syntax.UvmmProfiles, AttributeError: module 'univention.admin.syntax' has no attribute 'UvmmProfiles' Traceback (most recent call last): File "/usr/lib/univention-directory-listener/system/udm_extension.py", line 203, in handler udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 121, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 105, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/univention/admin/handlers/uvmm/profile.py", line 153, in <module> syntax=udm_syntax.UvmmCapacity, AttributeError: module 'univention.admin.syntax' has no attribute 'UvmmCapacity' Traceback (most recent call last): File "/usr/lib/univention-directory-listener/system/udm_extension.py", line 203, in handler udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 121, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 105, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/univention/admin/handlers/uvmm/profile.py", line 153, in <module> syntax=udm_syntax.UvmmCapacity, AttributeError: module 'univention.admin.syntax' has no attribute 'UvmmCapacity' Traceback (most recent call last): File "/usr/lib/univention-directory-listener/system/udm_extension.py", line 203, in handler udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 121, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 105, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/univention/admin/handlers/uvmm/profile.py", line 153, in <module> syntax=udm_syntax.UvmmCapacity, AttributeError: module 'univention.admin.syntax' has no attribute 'UvmmCapacity' Traceback (most recent call last): File "/usr/lib/univention-directory-listener/system/udm_extension.py", line 203, in handler udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 121, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 105, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked 21.02.21 00:50:09.115 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:10.551 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:11.927 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:12.760 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:13.674 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:14.763 LISTENER ( WARN ) : handler: udm_extension (failed) 21.02.21 00:50:15.950 LISTENER ( WARN ) : finished initializing module udm_extension with rv=0
(In reply to Florian Best from comment #9) > Before this happened it fails activating e.g. > cn=66univention-ldap-server_acl-master-uvmm,cn=ldapacl,cn=univention, > dc=autotest070,dc=local > due to Bug #52815. Maybe this also causes the UDM syntax to fail. Let's see > first if fixing Bug #52815 helps. This doesn't help.
07a1fdaa redirect check.sh output to logfile univention-updater 15.0.3-15A~5.0.0.202102231839 About comment 8 + 9 + 10: These are symptoms not caused by this change. See bug 515321 comment 4
(In reply to Erik Damrose from comment #11) > 07a1fdaa redirect check.sh output to logfile > univention-updater 15.0.3-15A~5.0.0.202102231839 > > About comment 8 + 9 + 10: These are symptoms not caused by this change. See > bug 515321 comment 4 Bug #51531 comment 4.
update on slave and member server fails with: Custom preupdate script /var/lib/local-preup.sh not found Update will wait here for 60 seconds... Press CTRL-c to abort or press ENTER to continue Checking disk_space ... OK Checking failed_ldif ... OK Checking hold_packages ... OK Checking kernel ... OK Checking ldap_connection ... OK Checking ldap_schema ... OK Checking legacy_objects ... Ignoring test as requested by update50/ignore_legacy_objects IGNORED Checking master_version ... OK Checking md5_signature_is_used ... OK Checking min_version ... OK Checking minimum_ucs_version_of_all_systems_in_domain ... OK Checking old_packages ... OK Checking overwritten_umc_templates ... OK Checking package_status ... OK Checking role_package_removed ... OK Checking slapd_on_member ... OK Checking ssh ... OK Checking system_date_too_old ... OK Checking system_role ... OK Checking term ... OK Checking usr_mountpoint ... OK Checking valid_machine_credentials ... OK preupNOn5nJ.sh: Cannot get LDAP credentials from '/etc/ldap.secret' Error: Update aborted by pre-update script of release 5.0-0 exitcode of univention-updater: 1 ERROR: update failed. Please check /var/log/univention/updater.log i guess we do not need that delete_legacy_objects delete_obsolete_objects stuff on members and slaves
(In reply to Florian Best from comment #12) > (In reply to Erik Damrose from comment #11) > > 07a1fdaa redirect check.sh output to logfile > > univention-updater 15.0.3-15A~5.0.0.202102231839 > > > > About comment 8 + 9 + 10: These are symptoms not caused by this change. See > > bug 515321 comment 4 > > Bug #51531 comment 4. I added a commit to suppress these errors from being logged ~10 times during the upgrade to UCS 5.0: git: a808e1211c29241f07a9f6388be9e036bd42c0c6 fix[uvmm]: prevent AttributeError when syntax class is not registered yet During a `univention-directory-listener-ctrl resync udm_extension` (on upgrade to UCS 5.0) all UDM extensions are loaded in arbitrary order. This causes failing import if a UDM module is handled before the required UDM syntax class exists. Therefore a fallback syntax class is used.
(In reply to Felix Botner from comment #13) > i guess we do not need that delete_legacy_objects delete_obsolete_objects > stuff on members and slaves The primary DC has to update first to UCS 5, deletion is only triggered on that server role. The fix was done on bug 51655, i am documenting the change here for completeness 3901eb66 Delete obsolete LDAP only on primary dc univention-updater 15.0.3-16A~5.0.0.202102241851
OK - ldap registration udm modules/syntax/po/ schema/acl OK - ucsversionend OK - update to 5.0 (master), uvmm still usable on (4.4) backup/slave OK - no errors during update to 5.0 OK - yaml
<https://errata.software-univention.de/#/?erratum=4.4x906>