Univention Bugzilla – Bug 48956
Enable Referential Integrity (refint) ldap overlay for consistency of LDAP references
Last modified: 2022-02-01 21:35:25 CET
We should enable the refint overlay module: https://www.openldap.org/doc/admin24/overlays.html 12.11. Referential Integrity overlay refint refint_attributes secretary,univentionNetworkLink,… refint_nothing "cn=admin,dc=example,dc=com" When we have this, whenever a modrdn or a remove on a DN is performed the references are updated everywhere! We don't need to handle these things anymore in UDM manually. I think we also have a chance that this works for group memberships. If yes, we can get rid of code like fast_member_remove().
This requires the selected attributes to have distinguishedName syntax (1.3.6.1.4.1.1466.115.121.1.12). univentionNetworkLink is not DN syntax :-/ Probably a lot more isn't.
Created attachment 9918 [details] patch This works very nice! I tested it with the secretary attributes of users. There is a sentence in the manpage I don't quite understand fully: Modifications performed by this overlay are not propagated during replication. This overlay must be configured identically on replication consumers in order to maintain full synchronization with the provider. Does this affect our DC Slaves?
I guess this is similar to the memberof overlay, the slapd does something behind your back. The manpage says that the OpenLDAP native replication (syncrepl) doesn't see these changes or at least doesn't propagate them. It is expected that the target system of the replication also runs refint and that does the same magic thing as the overlay on the source server. We would have to check if translog can see the changes or not.
*** Bug 26384 has been marked as a duplicate of this bug. ***
Activating this for uniqueMember on a UCS 5.0 system was surprisingly smooth. At least simple UDM operations like renaming a user didn't cause any issues. Re: Comment 3 I checked a simple ldapmordn -r and the transaction file even shows changes for the groups: 1327 uid=user6b,dc=ucs50domain,dc=net r 1328 uid=user6c,dc=ucs50domain,dc=net a 1329 cn=Domain Users,cn=groups,dc=ucs50domain,dc=net m 1330 cn=group1,cn=groups,dc=ucs50domain,dc=net m S4C was disabled and both groups then have: modifiersName: cn=Referential Integrity Overlay
Package: univention-ldap Version: 15.0.3-5A~4.4.0.202111241817 Branch: ucs_4.4-0 Scope: errata4.4-8 refint is enabled for the attribute uniqueMember (and that attribute only). Can be disabled with ucr set ldap/refint='false'. For now I leave it enabled even in a mere package update because I want it to run against our tests. If everything works, we can still disable it for updates.
Use case: Other modules that have (generic) "members" of some sort. To be investigated: * Does the listener trigger? * Can we remove logic in UDM?
Setting to FIXED just so that it pops up in certain queues. Investigation is still ongoing.
I wonder that we are suddenly do such kind of changes, nice! Why are we doing this now? I would prefer that the attributes are also configurable via UCR. In general it would be better to use this bug for UCS 5.0 and clone it for backports, so that we have this bug with more information in the git-history/blame. If it's configurable we cannot remove code from UDM. We should also consider UDM code running on a Slave in mixed environments (/ or during 5.0 upgrade) if we remove code from UDM now. My feeling says: better only remove UDM code in 5.0.
To be clear about this: The scope of this bug is to enable the refint-overlay only. Removing UDM code would be a separate bug entirely and should not be done in UCS 4.4. So the question here is, if we can enable this overlay without any regressions. The regression from Comment 10 points to an error in the UDM code, which seems to be fixed in UCS 5.0-0 via commit fa6de80fffd. So I propose backporting that fix.
Please provide performance metrics! How does enabling this module impact importing 100.000 students?
(In reply to Philipp Hahn from comment #13) > Please provide performance metrics! How does enabling this module impact > importing 100.000 students? (UCS 5) This is not the most scientific approach. But I added 1000 users: for i in $(seq 1 1000); do udm users/user create --position cn=users --set username=user$i --set lastname=Test --set password=univention --append groups=cn=g1,cn=groups,$(ucr get ldap/base); done I then renamed 10 of them: time for i in $(seq 1 10); do udm users/user modify --dn uid=user$i,cn=users,$(ucr get ldap/base) --set username=userx$i; done With refint and new UDM: real 1m55,874s user 0m1,026s sys 0m0,147s Without refint and old UDM: real 5m56,613s user 0m1,040s sys 0m0,152s We may want to look into UDM. Renaming 10 users (each in 2 groups) should not take 6 minutes and not 2 minutes.
Verified: * Code review * Package update * Functional & Performance tests (users/user, groups/group, computers/ubuntu) * ucr set ldap/refint; service slapd restart * Functional & Performance tests * Advisory
This can be activated via UCR variable "ldap/refint" - all other overlay modules are enabled via "ldap/overlay/...".
Created attachment 10877 [details] rename variable and add UCR variable for the refint attributes
QA: we are not setting `refint_nothing` - did you test what happens when removing the last member (of a non-primary-group)?
(In reply to Florian Best from comment #18) > QA: we are not setting `refint_nothing` - did you test what happens when > removing the last member (of a non-primary-group)? This is only relevant for a schema that absolutely requires any value to be set. Groups do not have this restriction, you may have empty groups.
(In reply to Florian Best from comment #16) > This can be activated via UCR variable "ldap/refint" - all other overlay > modules are enabled via "ldap/overlay/...". No, e.g., ldap/shadowbind or ldap/k5pwd. I would argue that the majority is actually the other way around.
OK. FYI: in UCS@school 5.0 I see the following traceback in Jenkins: Traceback (most recent call last): File "/usr/share/ucs-test/90_ucsschool/55_class_assignment.py", line 52, in schoolenv yield schoolenv File "/usr/lib/python3/dist-packages/univention/testing/ucsschool/ucs_test_school.py", line 213, in __exit__ self.cleanup() File "/usr/lib/python3/dist-packages/univention/testing/ucsschool/ucs_test_school.py", line 316, in cleanup self.cleanup_ou(ou_name, wait_for_replication=False) File "/usr/lib/python3/dist-packages/univention/testing/ucsschool/ucs_test_school.py", line 412, in cleanup_ou self.remove_dcs_from_global_groups(ou_name) File "/usr/lib/python3/dist-packages/univention/testing/ucsschool/ucs_test_school.py", line 471, in remove_dcs_from_global_groups self.lo.modify(group_dn, ml) File "/usr/lib/python3/dist-packages/univention/admin/uldap.py", line 814, in modify raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg) univention.admin.uexceptions.ldapError: No such attribute: modify/delete: uniqueMember: no such attribute ( Traceback (most recent call last): File "/usr/lib/python3/dist-packages/univention/testing/ucsschool/ucs_test_school.py", line 213, in __exit__ self.cleanup() File "/usr/lib/python3/dist-packages/univention/testing/ucsschool/ucs_test_school.py", line 316, in cleanup self.cleanup_ou(ou_name, wait_for_replication=False) File "/usr/lib/python3/dist-packages/univention/testing/ucsschool/ucs_test_school.py", line 412, in cleanup_ou self.remove_dcs_from_global_groups(ou_name) File "/usr/lib/python3/dist-packages/univention/testing/ucsschool/ucs_test_school.py", line 471, in remove_dcs_from_global_groups self.lo.modify(group_dn, ml) File "/usr/lib/python3/dist-packages/univention/admin/uldap.py", line 814, in modify raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg) univention.admin.uexceptions.ldapError: No such attribute: modify/delete: uniqueMember: no such attribute Traceback (most recent call last): File "/usr/lib/python3/dist-packages/univention/admin/uldap.py", line 803, in modify return self.lo.modify(dn, changes, serverctrls=serverctrls, response=response, rename_callback=rename_callback) File "/usr/lib/python3/dist-packages/univention/uldap.py", line 208, in _decorated return func(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/univention/uldap.py", line 753, in modify self.modify_ext_s(dn, ml, serverctrls=serverctrls, response=response) File "/usr/lib/python3/dist-packages/univention/uldap.py", line 208, in _decorated return func(self, *args, **kwargs) File "/usr/lib/python3/dist-packages/univention/uldap.py", line 812, in modify_ext_s rtype, rdata, rmsgid, resp_ctrls = self.lo.modify_ext_s(dn, ml, serverctrls=serverctrls) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1253, in modify_ext_s return self._apply_method_s(SimpleLDAPObject.modify_ext_s,*args,**kwargs) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1197, in _apply_method_s return func(self,*args,**kwargs) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 602, in modify_ext_s resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 749, in result3 resp_ctrl_classes=resp_ctrl_classes File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 756, in result4 ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop) File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 329, in _ldap_call reraise(exc_type, exc_value, exc_traceback) File "/usr/lib/python3/dist-packages/ldap/compat.py", line 44, in reraise raise exc_value File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 313, in _ldap_call result = func(*args,**kwargs) ldap.NO_SUCH_ATTRIBUTE: {'desc': 'No such attribute', 'info': 'modify/delete: uniqueMember: no such attribute'} )
<https://errata.software-univention.de/#/?erratum=4.4x1127> <https://errata.software-univention.de/#/?erratum=4.4x1128>