It seems you changed / introduced the joinscript "92univention-fetchmail-schema.inst" on 2023-07-12 (timestamp on my server). The Join script running this file fails after update to UCS 5.0-4.

The setup is a Primary DC and a Member Server with Kopano and fetchmail. Some users have their fetchmail configuration in the "old" style "Remote mail retrieval" without any single or multi drop.


Redacted output of relevant parts of /var/log/univention/join.log follows:

RUNNING 92univention-fetchmail.inst
2023-08-21 16:01:08.681314685+02:00 (in joinscript_init)
Object exists: cn=services,cn=univention,dc=mibes,dc=it
Object exists: cn=Fetchmail,cn=services,cn=univention,dc=mibes,dc=it
No modification: cn=xxxxx,cn=memberserver,cn=computers,dc=mibes,dc=it
WARNING: cannot append Fetchmail to service, value exists
Converting old fetchmail configuration...
Skip object with uid "xxxxx". Already migrated or incomplete configuration
Traceback (most recent call last):
  File "/usr/share/univention-fetchmail/migrate-fetchmail.py", line 189, in <module>
    main()
  File "/usr/share/univention-fetchmail/migrate-fetchmail.py", line 184, in main
    c.main()
  File "/usr/share/univention-fetchmail/migrate-fetchmail.py", line 104, in main
    ret = self.convert()
  File "/usr/share/univention-fetchmail/migrate-fetchmail.py", line 165, in convert
    ('univentionFetchmailUseSSL', ssl, []), ('univentionFetchmailSingle', old_fetchmail_new, map_fetchmail(updated_fetchmail_new)),
  File "/usr/share/univention-fetchmail/migrate-fetchmail.py", line 74, in map_fetchmail
    return [json.dumps(v).encode('UTF-8') for v in value]
  File "/usr/share/univention-fetchmail/migrate-fetchmail.py", line 74, in <listcomp>
    return [json.dumps(v).encode('UTF-8') for v in value]
  File "/usr/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

__JOINERR__:FAILED: /usr/lib/univention-install/92univention-fetchmail.inst
EXITCODE=1


After creating a test user with multiple single drop entries and adding debug prints in the the file /usr/share/univention-fetchmail/migrate-fetchmail.py, I found out that the function "map_fetchmail" should be rewritten, so that it produces data that is identical to the entries produced with the web frontend.

The ldap change call seems to expect an array of strings, which it converts into LDAP attributes. The "map_fetchmail" should transform the internal data into that format. It assumes, that every entry thing is already present as string, which is not true in this case. The entries are present as array of bytes (from conversion) or array of strings (decoded via unmap_fetchmail from present entries).

I tried to change the method converter.convert, as I hoped that would produce more elegant code, but that didn't work out. Therefore I just changed "map_fetchmail":

def map_fetchmail(value):
#    return [json.dumps(v).encode('UTF-8') for v in value]
    ret = []
    for elem in value:
        entry = []
        for param in elem:
            entry.append(param if isinstance(param, str) else param.decode())
        ret.append(json.dumps(entry).encode('UTF-8'))
    return ret


In my test setup it worked; I executed it directly and not via join scripts. I converted users with only "Remote mail retrieval" and a combination of "Remote mail retrieval (single drop)" and "Remote mail retrieval". This is just works-for-me, please verify the bug and see if this change fixes the bug as well.

Created attachment 11107 [details]
WiP-Version of my debug script without any warranty or fitness for work.

After a good night's sleep, I'll provide some more debugging information:

The format used by the web frontend to store the single drop mail retrieval is as follows:

root@xxxxx:~# univention-ldapsearch uid=abcde | grep univentionFetchmailSingle
univentionFetchmailSingle: ["mail.example.com", "IMAP", "abcde@mibes.it", "pw_redacted", "1", "1"]
univentionFetchmailSingle: ["mail.example.com", "POP", "cdefgh@mibes.it", "pw_redacted", "0", "0"]


In converter.convert, you fetch the data from ldap (line incomplete), which I debug printed:

attr=['uid', 'univentionFetchmailAddress', 'univentionFetchmailServer', 'univentionFetchmailProtocol', 'univentionFetchmailPasswd', 'univentionFetchmailKeepMailOnServer', 'univentionFetchmailUseSSL', 'univentionFetchmailSingle']
self.debug(attrs)

=> {'uid': [b'abcde'], 'univentionFetchmailProtocol': [b'IMAP'], 'univentionFetchmailSingle': [b'["mail.example.com", "IMAP", "abcde@mibes.it", "pw_redacted", "1", "1"]', b'["mail.example.com", "POP3", "cdefgh@mibes.it", "pw_redacted", "0", "0"]'], 'univentionFetchmailServer': [b'mail.example.com'], 'univentionFetchmailAddress': [b'efghi@mibes.it'], 'univentionFetchmailPasswd': [b'pw_redacted'], 'univentionFetchmailUseSSL': [b'1']}


Before creating the ldap change, I print out some debug messages. First the current value of univentionFetchmailSingle LDAP entry (an array of two binary-strings, which are in json format, each holding an array):

self.debug(old_fetchmail_new)

=> [b'["mail.example.com", "IMAP", "abcde@mibes.it", "pw_redacted", "1", "1"]', b'["mail.example.com", "POP3", "cdefg@mibes.it", "pw_redacted", "0", "0"]']


Then updated fetchmail_new, which holds the data of the old entries and the newly created entry in the internal data representation (array of entries, each entry is itself an array, either of strings or a binary-strings):

self.debug(updated_fetchmail_new)

=> [['mail.example.com', 'IMAP', 'abcde@mibes.it', 'pw_redacted', '1', '1'], ['mail.example.com', 'POP3', 'cdefg@mibes.it', 'pw_redacted', '0', '0'], [b'mail.example.com', b'IMAP', b'efghi@mibes.it', b'pw_redacted', b'0', b'1']]


And then finally the data in the format that is given to the ldap modify command (array of binary-strings, which are in json format, each holding an array), which matches the input at the beginning of this clarification:

self.debug(map_fetchmail(updated_fetchmail_new))

=> [b'["mail.example.com", "IMAP", "abcde@mibes.it", "pw_redacted", "1", "1"]', b'["mail.example.com", "POP3", "cdefg@mibes.it", "pw_redacted", "0", "0"]', b'["mail.example.com", "IMAP", "efghi@mibes.it", "pw_redacted", "0", "1"]']


That concludes my debugging session; attached is my "debug version" of the script for your reference. Feel free to do with that code whatever you like, I would give it to you (copyright and stuff).

Comment 5 Juan Carlos 2023-09-11 08:44:38 CEST

Changes:

- Increased univention-fetchmail-schema joinscript version
- Fixed migrate-fetchmail.py script to handle bytes correctly.

Commits:

univention-fetchmail (13.0.7-1)
417f5763d957 | Bug #56482: Changelog

univention-fetchmail (13.0.6-5)
db773b5b92a3 | Bug #56308: Increase joinscript version
15f4113eb21e | Bug #56308: handle bytes and str on fetchmail migration script and hook


New version:

Package: univention-fetchmail
Version: 13.0.7-1
Branch: ucs_5.0-0
Scope: ucs5.0-5

Comment 6 Juan Carlos 2023-09-11 12:17:31 CEST

New version:

Package: univention-fetchmail
Version: 13.0.7-2
Branch: ucs_5.0-0
Scope: ucs5.0-5

Comment 7 Dirk Wiesenthal 2023-09-20 09:00:20 CEST

UCS 5.0-5 has been released