Bug 40485 - AppCenter: wrong encoding causes crash
AppCenter: wrong encoding causes crash
Status: REOPENED
Product: UCS
Classification: Unclassified
Component: UMC - App-Center
UCS 5.0
Other Linux
: P5 normal (vote)
: ---
Assigned To: UMC maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-22 10:58 CET by Florian Best
Modified: 2021-12-17 11:14 CET (History)
5 users (show)

See Also:
What kind of report is it?: Development Internal
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2021121621000575, 2019042621000709, 2018092821000994, 2018061121000057, 2018051521001327, 2018051621001512, 2018022121000161, 2017090721000507, 2020030821000046, 2020112621000014
Bug group (optional): Error handling, External feedback
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Florian Best univentionstaff 2016-01-22 10:58:09 CET
Re received the following traceback, 4.1-0 errata14 (Vahr).

Execution of command 'appcenter/query' has failed:

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 283, in execute
    function(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 320, in _response
    self.finished(request.id, result[0])
  File "%PY2.7%/univention/management/console/base.py", line 401, in finished
    self.result(res)
  File "%PY2.7%/univention/management/console/base.py", line 405, in result
    self.signal_emit('success', response)
  File "%PY2.7%/notifier/signals.py", line 75, in signal_emit
    self.__signals[ signal ].emit( *args )
  File "%PY2.7%/notifier/signals.py", line 41, in emit
    cb( *args )
  File "%PY2.7%/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 118, in _reply
    self.response(msg)
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 350, in response
    self.__queue += str(msg)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 337, in __str__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 130, in _formattedMessage
    data = json.dumps(body)
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 321, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 237, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 311, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xae in position 9: invalid start byte

It seems that there was a ® which was not encoded in UTF-8 (\xc2\xae) but in latin-1 (\xae) which causes this error.
Comment 1 Dirk Wiesenthal univentionstaff 2016-02-22 09:46:01 CET
Probably best to prevent this on our App Center server, not ignoring it (or guessing the encoding) on the UCS server side.
Comment 2 Dirk Wiesenthal univentionstaff 2016-10-28 13:38:16 CEST
We now have tools on the App Center server to prevent this.
Comment 3 Florian Best univentionstaff 2017-09-22 18:10:49 CEST
(In reply to Dirk Wiesenthal from comment #2)
> We now have tools on the App Center server to prevent this.
But it happened again.

Version: 4.2-2 errata159 (Lesum)

Remark: Not able to install via UCS - Will not fetch docker image.... Owncloud installs, but I did not want OC, but I suppose I will have to.
Comment 4 Johannes Keiser univentionstaff 2018-04-27 12:45:32 CEST
Reported again: Version: 4.2-3 errata305 (Lesum)
Comment 5 Johannes Keiser univentionstaff 2018-05-17 14:19:23 CEST
Reported again:  Version: 4.3-0 errata83 (Neustadt)

Interner Server-Fehler in "appcenter/packages/query".
Request: appcenter/packages/query

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 253, in execute
    function.__func__(self, request, *args, **kwargs)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 192, in _response
    return function(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 320, in _response
    self.finished(request.id, result[0])
  File "%PY2.7%/univention/management/console/base.py", line 434, in finished
    self.result(res)
  File "%PY2.7%/univention/management/console/base.py", line 438, in result
    self.signal_emit('success', response)
  File "%PY2.7%/notifier/signals.py", line 75, in signal_emit
    self.__signals[ signal ].emit( *args )
  File "%PY2.7%/notifier/signals.py", line 41, in emit
    cb( *args )
  File "%PY2.7%/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 124, in _reply
    self.response(msg)
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 354, in response
    self.__queue += str(msg)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 308, in __str__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 114, in _formattedMessage
    data = json.dumps(body)
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 380, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 373, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 7: invalid start byte
Comment 6 Johannes Keiser univentionstaff 2018-05-17 14:22:02 CEST
Reported again: Version: 4.3-0 errata35 (Neustadt)
Same Traceback as Comment #5
Comment 7 Johannes Keiser univentionstaff 2018-07-30 17:39:01 CEST
Version: 4.3-0 errata112 (Neustadt)

Internal server error during "appcenter/docker/progress".
Request: appcenter/docker/progress

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 253, in execute
    function.__func__(self, request, *args, **kwargs)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 320, in _response
    self.finished(request.id, result[0])
  File "%PY2.7%/univention/management/console/base.py", line 434, in finished
    self.result(res)
  File "%PY2.7%/univention/management/console/base.py", line 438, in result
    self.signal_emit('success', response)
  File "%PY2.7%/notifier/signals.py", line 75, in signal_emit
    self.__signals[ signal ].emit( *args )
  File "%PY2.7%/notifier/signals.py", line 41, in emit
    cb( *args )
  File "%PY2.7%/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 124, in _reply
    self.response(msg)
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 354, in response
    self.__queue += str(msg)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 308, in __str__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 114, in _formattedMessage
    data = json.dumps(body)
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 380, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 373, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 13: invalid continuation byte
Comment 8 Johannes Keiser univentionstaff 2018-10-10 19:19:04 CEST
Reported again: Version: 4.3-2 errata229 (Neustadt)

Interner Server-Fehler in "setup/apps/query (wizard)".
Request: setup/apps/query (wizard)

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 253, in execute
    function.__func__(self, request, *args, **kwargs)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 318, in _response
    result = _multi_response(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 192, in _response
    return function(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 440, in _response
    return list(function(self, iterator, *nones))
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 286, in _fake_func
    yield function(self, *args)
  File "%PY2.7%/univention/management/console/modules/setup/__init__.py", line 749, in apps_query
    return util.get_apps(True)
  File "%PY2.7%/univention/management/console/modules/setup/util.py", line 790, in get_apps
    return [get.to_dict(app) for app in Apps().get_all_apps() if app.is_ucs_component()]
  File "%PY2.7%/univention/appcenter/app_cache.py", line 119, in get_all_apps
    for app in self.get_every_single_app():
  File "%PY2.7%/univention/appcenter/app_cache.py", line 533, in get_every_single_app
    for app in app_cache.get_every_single_app():
  File "%PY2.7%/univention/appcenter/app_cache.py", line 492, in get_every_single_app
    ret.extend(app_cache.get_every_single_app())
  File "%PY2.7%/univention/appcenter/app_cache.py", line 353, in get_every_single_app
    if self._save_cache():
  File "%PY2.7%/univention/appcenter/app_cache.py", line 214, in _save_cache
    dump([app.attrs_dict() for app in self._cache], fd, indent=2)
  File "/usr/lib/python2.7/json/__init__.py", line 189, in dump
    for chunk in iterable:
  File "/usr/lib/python2.7/json/encoder.py", line 431, in _iterencode
    for chunk in _iterencode_list(o, _current_indent_level):
  File "/usr/lib/python2.7/json/encoder.py", line 332, in _iterencode_list
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 390, in _iterencode_dict
    yield _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc3 in position 4: invalid continuation byte
Comment 9 Stefan Gohmann univentionstaff 2019-01-03 07:18:42 CET
This issue has been filled against UCS 4.1. The maintenance with bug and security fixes for UCS 4.1 has ended on 5st of April 2018.

Customers still on UCS 4.1 are encouraged to update to UCS 4.3. Please contact
your partner or Univention for any questions.

If this issue still occurs in newer UCS versions, please use "Clone this bug" or simply reopen the issue. In this case please provide detailed information on how this issue is affecting you.
Comment 10 Johannes Keiser univentionstaff 2019-05-14 15:47:32 CEST
Reported again: Version: 4.4-0 errata59 (Blumenthal)

Interner Server-Fehler in "appcenter/packages/query".
Request: appcenter/packages/query

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 253, in execute
    function.__func__(self, request, *args, **kwargs)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 192, in _response
    return function(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 320, in _response
    self.finished(request.id, result[0])
  File "%PY2.7%/univention/management/console/base.py", line 462, in finished
    self.result(res)
  File "%PY2.7%/univention/management/console/base.py", line 466, in result
    self.signal_emit('success', response)
  File "%PY2.7%/notifier/signals.py", line 75, in signal_emit
    self.__signals[ signal ].emit( *args )
  File "%PY2.7%/notifier/signals.py", line 41, in emit
    cb( *args )
  File "%PY2.7%/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 128, in _reply
    self.response(msg)
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 359, in response
    self.__queue += str(msg)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 304, in __str__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 110, in _formattedMessage
    data = json.dumps(body)
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 7: invalid start byte
Comment 11 Christian Castens univentionstaff 2020-05-14 13:02:28 CEST
Reported again: 4.4-3 errata385 (Blumenthal)

Internal server error during "appcenter/docker/progress".
Request: appcenter/docker/progress

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 358, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 261, in execute
    function.__func__(self, request, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/modules/decorators.py", line 323, in _response
    self.finished(request.id, result[0])
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 478, in finished
    self.result(res)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/base.py", line 482, in result
    self.signal_emit('success', response)
  File "%PY2.7%/notifier/signals.py", line 75, in signal_emit
    self.__signals[ signal ].emit( *args )
  File "%PY2.7%/notifier/signals.py", line 41, in emit
    cb( *args )
  File "%PY2.7%/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/modserver.py", line 130, in _reply
    self.response(msg)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/modserver.py", line 361, in response
    self.__queue += str(msg)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/message.py", line 305, in __str__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "/usr/lib/python2.7/dist-packages/univention/management/console/protocol/message.py", line 111, in _formattedMessage
    data = json.dumps(body)
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa9 in position 13: invalid start byte

Role: domaincontroller_master
Comment 12 Maximilian Janßen univentionstaff 2021-12-17 11:14:25 CET
Version: 5.0-1 errata175

Error: 
Internal server error during "appcenter/packages/query".
Request: appcenter/packages/query

Traceback (most recent call last):
  File "%PY3%/univention/management/console/base.py", line 344, in __error_handling
    six.reraise(etype, exc, etraceback)
  File "%PY3%/six.py", line 693, in reraise
    raise value
  File "%PY3%/univention/management/console/base.py", line 247, in execute
    function.__func__(self, request, *args, **kwargs)
  File "%PY3%/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "%PY3%/univention/management/console/modules/decorators.py", line 321, in _response
    result = _multi_response(self, request)
  File "%PY3%/univention/management/console/modules/decorators.py", line 181, in _response
    return function(self, request)
  File "%PY3%/univention/management/console/modules/decorators.py", line 443, in _response
    return list(function(self, iterator, *nones))
  File "%PY3%/univention/management/console/modules/decorators.py", line 289, in _fake_func
    yield function(self, *args)
  File "%PY3%/univention/management/console/modules/appcenter/__init__.py", line 789, in packages_query
    result.append(self._package_to_dict(package, full=False))
  File "%PY3%/univention/management/console/modules/appcenter/__init__.py", line 906, in _package_to_dict
    'summary': candidate.summary,
  File "%PY3%/apt/package.py", line 606, in summary
    return records.short_desc if records is not None else None
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 7: invalid start byte

Role: domaincontroller_master