Univention Bugzilla – Bug 54587
[ADC] Sqlite3 database from UCS 4.4 contains bytestrings
Last modified: 2023-05-24 15:43:39 CEST
+++ This bug was initially created as a clone of Bug #54586 +++ When values in the sqlite3 database of UCS 4.4 are used in UCS 5.0 (e.g. rejects) the sqlite3 engine returns them as bytestrings instead oder str/unicode. This leads to followup errors like: 24.03.2022 12:42:29.853 LDAP (PROCESS): Internal group membership cache was created 24.03.2022 12:42:29.968 LDAP (INFO ): Override identify function for container_dc 24.03.2022 12:42:29.971 LDAP (INFO ): sync UCS > AD: polling 24.03.2022 12:42:29.971 LDAP (PROCESS): sync AD > UCS: Resync rejected dn: 'CN=dns,DC=***' 24.03.2022 12:42:29.972 LDAP (ERROR ): unexpected Error during s4.resync_rejected 24.03.2022 12:42:29.972 LDAP (ERROR ): Traceback (most recent call last): File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 1814, in resync_rejected elements = self.__search_ad_changeUSN(change_usn, show_deleted=True) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 981, in __search_ad_changeUSN usn_filter = format_escaped('(|(uSNChanged={0!e})(uSNCreated={0!e}))', changeUSN) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 489, in format_escaped return LDAPEscapeFormatter().format(format_string, *args, **kwargs) File "/usr/lib/python3.7/string.py", line 186, in format return self.vformat(format_string, args, kwargs) File "/usr/lib/python3.7/string.py", line 190, in vformat result, _ = self._vformat(format_string, args, kwargs, used_args, 2) File "/usr/lib/python3.7/string.py", line 234, in _vformat obj = self.convert_field(obj, conversion) File "/usr/lib/python3/dist-packages/univention/s4connector/s4/__init__.py", line 474, in convert_field raise TypeError('Filter must be string, not bytes: %r' % (value,)) TypeError: Filter must be string, not bytes: b'3853' We could fix this via scripts like the following in the `postup.sh` of the UCS 5.0 update (before the S4-connector is started again): ``` #!/usr/bin/python3 import sqlite3 d = sqlite3.connect('/etc/univention/connector/s4internal.sqlite') c = d.cursor() c.execute('select * from "S4 rejected";') for k, v in c.fetchall(): if isinstance(v, bytes): v = v.decode('UTF-8') if isinstance(k, bytes): k = k.decode('UTF-8') c.execute('delete from "S4 rejected"') c.execute('insert into "S4 rejected" (key, value) VALUES (?, ?)', (k, v)) d.commit() ```
https://git.knut.univention.de/univention/ucs/-/merge_requests/740 https://git.knut.univention.de/univention/ucs/-/merge_requests/741
The script was wrong because it removed all rejects except for the last one. Fixed script is: systemctl stop univention-ad-connector python3 - <<EOF #!/usr/bin/python3 import sqlite3 db = sqlite3.connect('/etc/univention/connector/internal.sqlite') cursor = db.cursor() cursor.execute('select * from "AD rejected";') rejects = cursor.fetchall() cursor.execute('delete from "AD rejected"') for key, value in rejects: if isinstance(value, bytes): value = value.decode('UTF-8') if isinstance(key, bytes): key = key.decode('UTF-8') cursor.execute('insert into "AD rejected" (key, value) VALUES (?, ?)', (key, value)) db.commit() EOF systemctl start univention-ad-connector
The sqlite database is now migrated during the UCS 5.0 postup.sh update and during the UCS 5.0-3-errata update. UCS 5.0-3: univention-ad-connector.yaml cd7edcf38ba1 | chore(univention-s4-connector): update advisory univention-ad-connector (14.0.13-5) bed3fbaf68b8 | Bug #54587: replace bytestrings in AD-Connector reject table which UCS 5.0-0: f2f53a89b008 | Bug #54587: Bug #54586: replace bytestrings in AD/S4-Connector reject table after the upgrade to UCS 5.0-0
Verified: * Code review * Package update * Advisory
Support reported a special case, where the loops aborts with > sqlite3.IntegrityError: UNIQUE constraint failed: S4 rejected.Key See https://pastebin.knut.univention.de/oWcWkbsZ for details.
univention-ad-connector (14.0.13-5) b38aa1c6f326 | fixup! Bug #54587: replace bytestrings in AD-Connector reject table which were leftover from UCS 4.4 upgrade univention-updater (15.0.3-71) 1e9515e8a3ff | fixup! Bug #54587: Bug #54586: replace bytestrings in AD/S4-Connector reject table after the upgrade to UCS 5.0-0
<https://errata.software-univention.de/#/?erratum=5.0x677>