Univention Bugzilla – Bug 54586
[S4C] Sqlite3 database from UCS 4.4 contains bytestrings
Last modified: 2023-05-24 15:43:38 CEST
When values in the sqlite3 database of UCS 4.4 are used in UCS 5.0 (e.g. rejects) the sqlite3 engine returns them as bytestrings instead oder str/unicode. This leads to followup errors like: 24.03.2022 12:42:29.853 LDAP (PROCESS): Internal group membership cache was created 24.03.2022 12:42:29.968 LDAP (INFO ): Override identify function for container_dc 24.03.2022 12:42:29.971 LDAP (INFO ): sync UCS > AD: polling 24.03.2022 12:42:29.971 LDAP (PROCESS): sync AD > UCS: Resync rejected dn: 'CN=dns,DC=***' 24.03.2022 12:42:29.972 LDAP (ERROR ): unexpected Error during s4.resync_rejected 24.03.2022 12:42:29.972 LDAP (ERROR ): Traceback (most recent call last): File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 1814, in resync_rejected elements = self.__search_ad_changeUSN(change_usn, show_deleted=True) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 981, in __search_ad_changeUSN usn_filter = format_escaped('(|(uSNChanged={0!e})(uSNCreated={0!e}))', changeUSN) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 489, in format_escaped return LDAPEscapeFormatter().format(format_string, *args, **kwargs) File "/usr/lib/python3.7/string.py", line 186, in format return self.vformat(format_string, args, kwargs) File "/usr/lib/python3.7/string.py", line 190, in vformat result, _ = self._vformat(format_string, args, kwargs, used_args, 2) File "/usr/lib/python3.7/string.py", line 234, in _vformat obj = self.convert_field(obj, conversion) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 474, in convert_field raise TypeError('Filter must be string, not bytes: %r' % (value,)) TypeError: Filter must be string, not bytes: b'3853' We could fix this via scripts like the following in the `postup.sh` of the UCS 5.0 update (before the S4-connector is started again): ``` #!/usr/bin/python3 import sqlite3 d = sqlite3.connect('/etc/univention/connector/s4internal.sqlite') c = d.cursor() c.execute('select * from "S4 rejected";') for k, v in c.fetchall(): if isinstance(v, bytes): v = v.decode('UTF-8') if isinstance(k, bytes): k = k.decode('UTF-8') c.execute('delete from "S4 rejected"') c.execute('insert into "S4 rejected" (key, value) VALUES (?, ?)', (k, v)) d.commit() ```
Another customer effected Ticket#2023012421000522 UCS: 5.0-2 errata556 Installed: samba4=4.16 self-service=5.0 self-service-backend=5.0 Upgradable:
Another customer is affected Ticket#2023040421000297 UCS: 5.0-3 errata642 Installed: cups=2.2.1 dhcp-server=12.0 radius=5.0 samba4=4.16 squid=3.5 ucsschool=5.0 v3 4.4/ucsschool-veyon-proxy=4.7.4.14-0 Upgradable: samba4/role: DC server/role: domaincontroller_slave system/setup/boot/select/role: true 18.04.2023 16:14:16.995 LDAP (ERROR ): unexpected Error during s4.resync_rejected 18.04.2023 16:14:16.996 LDAP (ERROR ): Traceback (most recent call last): File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 1760, in resync_rejected elements = self.__search_ad_changeUSN(change_usn, show_deleted=True) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 965, in __search_ad_changeUSN usn_filter = format_escaped('(|(uSNChanged={0!e})(uSNCreated={0!e}))', changeUSN) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 489, in format_escaped return LDAPEscapeFormatter().format(format_string, *args, **kwargs) File "/usr/lib/python3.7/string.py", line 186, in format return self.vformat(format_string, args, kwargs) File "/usr/lib/python3.7/string.py", line 190, in vformat result, _ = self._vformat(format_string, args, kwargs, used_args, 2) File "/usr/lib/python3.7/string.py", line 234, in _vformat obj = self.convert_field(obj, conversion) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 474, in convert_field raise TypeError('Filter must be string, not bytes: %r' % (value,)) TypeError: Filter must be string, not bytes: b'822857'
https://git.knut.univention.de/univention/ucs/-/merge_requests/740 https://git.knut.univention.de/univention/ucs/-/merge_requests/741
The script was wrong because it removed all rejects except for the last one. Fixed script is: systemctl stop univention-s4-connector python3 - <<EOF #!/usr/bin/python3 import sqlite3 db = sqlite3.connect('/etc/univention/connector/s4internal.sqlite') cursor = db.cursor() cursor.execute('select * from "S4 rejected";') rejects = cursor.fetchall() cursor.execute('delete from "S4 rejected"') for key, value in rejects: if isinstance(value, bytes): value = value.decode('UTF-8') if isinstance(key, bytes): key = key.decode('UTF-8') cursor.execute('insert into "S4 rejected" (key, value) VALUES (?, ?)', (key, value)) db.commit() EOF systemctl start univention-s4-connector
The sqlite database is now migrated during the UCS 5.0 postup.sh update and during the UCS 5.0-3-errata update. UCS 5.0-3: univention-s4-connector.yaml 4dcd69d7b69a | chore(univention-s4-connector): update advisory univention-s4-connector (14.0.13-5) 05e1b9ec407c | Bug #54586: replace bytestrings in S4-Connector reject table which UCS 5.0: univention-updater (15.0.3-71) f2f53a89b008 | Bug #54587: Bug #54586: replace bytestrings in AD/S4-Connector reject table after the upgrade to UCS 5.0-0
Verified: * Code review * Package update * Advisory Not-yet verified: * postup.sh
univention-s4-connector (14.0.13-5) d876ee0840c8 | fixup! Bug #54586: replace bytestrings in S4-Connector reject table which were leftover from UCS 4.4 upgrade univention-updater (15.0.3-71) 1e9515e8a3ff | fixup! Bug #54587: Bug #54586: replace bytestrings in AD/S4-Connector reject table after the upgrade to UCS 5.0-0
After the errata release we should also activate the new postup.sh
<https://errata.software-univention.de/#/?erratum=5.0x675>