Bug 51290 - UnicodeDecodeError: 'utf8' codec can't decode byte in position x : invalid start byte
UnicodeDecodeError: 'utf8' codec can't decode byte in position x : invalid st...
Status: NEW
Product: UCS
Classification: Unclassified
Component: App Center
UCS 4.4
Other Linux
: P5 normal (vote)
: ---
Assigned To: App Center maintainers
App Center maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-05-15 09:24 CEST by Christian Castens
Modified: 2021-07-13 13:39 CEST (History)
2 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2020040921000882, 2020092721000026, 2020102221000747, 2020102221000729
Bug group (optional): Error handling, External feedback
Max CVSS v3 score:


Attachments
debug patch (747 bytes, patch)
2021-07-13 11:19 CEST, Florian Best
Details | Diff
workaround patch (726 bytes, patch)
2021-07-13 12:43 CEST, Florian Best
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Christian Castens univentionstaff 2020-05-15 09:24:46 CEST
Version: 4.4-3 errata413 (Blumenthal)

Internal server error during "appcenter/invoke_dry_run".
Request: appcenter/invoke_dry_run

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 358, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 261, in execute
    function.__func__(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/appcenter/__init__.py", line 398, in invoke_dry_run
    self.invoke(request)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/appcenter/__init__.py", line 127, in _deferred
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/decorators.py", line 639, in _decorated
    return function(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/appcenter/__init__.py", line 532, in invoke
    app = Apps().find(app_id)
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/app_cache.py", line 84, in find
    apps = self.get_all_apps_with_id(app_id)
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/app_cache.py", line 71, in get_all_apps_with_id
    for app in self.get_every_single_app():
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/app_cache.py", line 542, in get_every_single_app
    for app in app_cache.get_every_single_app():
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/app_cache.py", line 501, in get_every_single_app
    ret.extend(app_cache.get_every_single_app())
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/app_cache.py", line 362, in get_every_single_app
    if self._save_cache():
  File "/usr/lib/python2.7/dist-packages/univention/appcenter/app_cache.py", line 223, in _save_cache
    dump([app.attrs_dict() for app in self._cache], fd, indent=2)
  File "/usr/lib/python2.7/json/__init__.py", line 189, in dump
    for chunk in iterable:
  File "/usr/lib/python2.7/json/encoder.py", line 431, in _iterencode
    for chunk in _iterencode_list(o, _current_indent_level):
  File "/usr/lib/python2.7/json/encoder.py", line 332, in _iterencode_list
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 390, in _iterencode_dict
    yield _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 8: invalid start byte

Role: domaincontroller_master
Comment 1 Christian Castens univentionstaff 2020-09-30 15:50:40 CEST
reported again:
Version: 4.4-5 errata703 (Blumenthal)

Error:
Internal server error during "setup/apps/query (wizard)".
Request: setup/apps/query (wizard)

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 359, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "%PY2.7%/univention/management/console/base.py", line 262, in execute
    function.__func__(self, request, *args, **kwargs)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 321, in _response
    result = _multi_response(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 443, in _response
    return list(function(self, iterator, *nones))
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 289, in _fake_func
    yield function(self, *args)
  File "%PY2.7%/univention/management/console/modules/setup/__init__.py", line 759, in apps_query
    return util.get_apps(True)
  File "%PY2.7%/univention/management/console/modules/setup/util.py", line 792, in get_apps
    return [get.to_dict(app) for app in Apps().get_all_apps() if app.is_ucs_component()]
  File "%PY2.7%/univention/appcenter/app_cache.py", line 129, in get_all_apps
    for app in self.get_every_single_app():
  File "%PY2.7%/univention/appcenter/app_cache.py", line 543, in get_every_single_app
    for app in app_cache.get_every_single_app():
  File "%PY2.7%/univention/appcenter/app_cache.py", line 502, in get_every_single_app
    ret.extend(app_cache.get_every_single_app())
  File "%PY2.7%/univention/appcenter/app_cache.py", line 363, in get_every_single_app
    if self._save_cache():
  File "%PY2.7%/univention/appcenter/app_cache.py", line 224, in _save_cache
    dump([app.attrs_dict() for app in self._cache], fd, indent=2)
  File "/usr/lib/python2.7/json/__init__.py", line 189, in dump
    for chunk in iterable:
  File "/usr/lib/python2.7/json/encoder.py", line 431, in _iterencode
    for chunk in _iterencode_list(o, _current_indent_level):
  File "/usr/lib/python2.7/json/encoder.py", line 332, in _iterencode_list
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 390, in _iterencode_dict
    yield _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8d in position 0: invalid start byte

Role: None
Comment 2 stefan.bauer 2021-07-13 10:39:32 CEST
Same problem here. Latest UCS 4.4-8 after trying to edit a user in UCS management console. Nothing obvious wrong in user settings.

excerpt from ldap for specific user:

dn: uid=Mustermann_U,ou=Benutzer,ou=Fertigung,ou=Abteilungen,ou=musterfirma,dc=musterfirma,dc=local
uid: Mustermann_U
krb5PrincipalName: Mustermann_U@musterfirma.LOCAL
objectClass: krb5KDCEntry
objectClass: person
objectClass: automount
objectClass: top
objectClass: pkiUser
objectClass: krb5Principal
objectClass: organizationalPerson
objectClass: univentionPWHistory
objectClass: univentionMail
objectClass: inetOrgPerson
objectClass: shadowAccount
objectClass: sambaSamAccount
objectClass: posixAccount
objectClass: univentionObject
uidNumber: 2444
sambaAcctFlags: [U          ]
sambaPasswordHistory: I DELETED THIS FIELD
sambaBadPasswordCount: 0
krb5MaxLife: 86400
cn: Uwe Mustermann
mailForwardCopyToSelf: 0
krb5MaxRenew: 604800
gecos: Uwe Mustermann
sambaBadPasswordTime: 0
userCertificate;binary:: I DELETED THIS FIELD - WAS BINARY
description: RND
loginShell: /bin/bash
univentionObjectType: users/user
krb5KDCFlags: 126
gidNumber: 5001
sambaPrimaryGroupSID: S-1-5-21-588273740-1646099605-1082013118-513
displayName: Uwe Mustermann
mailPrimaryAddress: uwe.Mustermann@musterfirmamuc.tld
sambaHomeDrive: H:
l: Muenchen
sambaSID: S-1-5-21-588273740-1646099605-1082013118-1117
sn: Mustermann
pwhistory: I DELETED THIS FIELD
homeDirectory: /home/Mustermann_U
givenName: Uwe
sambaNTPassword: I DELETED THIS FIELD
krb5Key:: I DELETED THIS FIELD
krb5Key:: I DELETED THIS FIELD
krb5Key:: I DELETED THIS FIELD
krb5Key:: I DELETED THIS FIELD
krb5KeyVersionNumber: 11
userPassword:: I DELETED THIS FIELD
shadowLastChange: 18778
sambaPwdLastSet: 1622439430

------------- error message --------------------------------------

  File "/usr/sbin/univention-management-console-module", line 120, in <module>
    notifier.loop()
  File "/usr/lib/python2.7/dist-packages/notifier/nf_generic.py", line 304, in loop
    step()
  File "/usr/lib/python2.7/dist-packages/notifier/nf_generic.py", line 295, in step
    __min_timer = dispatch.dispatcher_run()
  File "/usr/lib/python2.7/dist-packages/notifier/dispatch.py", line 75, in dispatcher_run
    if not disp():
  File "/usr/lib/python2.7/dist-packages/notifier/threads.py", line 161, in _simple_threads_dispatcher
    task.announce()
  File "/usr/lib/python2.7/dist-packages/notifier/threads.py", line 139, in announce
    self._callback(self, self._result)
  File "/usr/lib/python2.7/dist-packages/notifier/__init__.py", line 104, in __call__
    return self._function(*tmp, **self._kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 298, in thread_finished_callback
    self.finished(request.id, result)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 490, in finished
    self.result(res)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 494, in result
    self.signal_emit('success', response)
  File "/usr/lib/python2.7/dist-packages/notifier/signals.py", line 84, in signal_emit
    self.__signals[signal].emit(*args)
  File "/usr/lib/python2.7/dist-packages/notifier/signals.py", line 49, in emit
    cb(*args)
  File "/usr/lib/python2.7/dist-packages/notifier/__init__.py", line 104, in __call__
    return self._function(*tmp, **self._kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/modserver.py", line 141, in _reply
    self.response(msg)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/modserver.py", line 378, in response
    self.__queue += bytes(msg)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/message.py", line 140, in __str__
    return self.__bytes__()
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/message.py", line 342, in __bytes__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/message.py", line 122, in _formattedMessage
    data = json.dumps(body)
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 13: invalid start byte
Comment 3 Florian Best univentionstaff 2021-07-13 11:19:13 CEST
Created attachment 10784 [details]
debug patch

Hello Stefan,

can you apply this patch to the system, execute "systemctl restart univention-management-console-server", retry the action and paste the output here.

patch -i /root/51290.patch -d / -p0
Comment 4 stefan.bauer 2021-07-13 11:26:50 CEST
Thank you.

Will send output via e-mail as it contains sensitive data.
Comment 5 Florian Best univentionstaff 2021-07-13 11:59:41 CEST
(In reply to stefan.bauer from comment #4)
> Thank you.
> 
> Will send output via e-mail as it contains sensitive data.
Thank you.
The problem is the property 'certificateIssuerLocation' which has a value similar to 'F\x00l\x00o\x00r\x00i\x00\xe4\x00n' (I replaced the name with my name).
Comment 6 stefan.bauer 2021-07-13 12:13:57 CEST
Thank you very much for your analysis. How can i modify this field from command-line? Is this a required field?
Comment 7 Florian Best univentionstaff 2021-07-13 12:43:52 CEST
Created attachment 10785 [details]
workaround patch

(In reply to stefan.bauer from comment #6)
> Thank you very much for your analysis. How can i modify this field from
> command-line? Is this a required field?

It is part of the user certificate. It's not required for a user object, but I don't know about your environment.

You can either reupload a fixed new certificate:

udm users/user modify --dn "$DN" --set userCertificate="$(base64 < userCertificate.der)"

Or remove the currently set one:
udm users/user modify --dn "$DN" --remove userCertificate

Or you can upgrade to UCS 5.0 because it looks like this won't happen in UCS 5.0 (but I did not test it).

Or you can temporarily patch your environment with the attached patch:

patch -i /root/51290-2.patch -d / -p0
Comment 8 stefan.bauer 2021-07-13 13:38:58 CEST
Awesome. Thank you a lot! That did the trick :)
Comment 9 stefan.bauer 2021-07-13 13:39:43 CEST
for the record, we removed the certs from the affected users.