Bug 40882 - TypeError: appcenter.app.AppAttribute is not JSON serializable
TypeError: appcenter.app.AppAttribute is not JSON serializable
Product: UCS
Classification: Unclassified
Component: UMC - App-Center
UCS 4.1
Other Linux
: P5 normal (vote)
: UCS 4.1-1-errata
Assigned To: Dirk Wiesenthal
Florian Best
Depends on:
  Show dependency treegraph
Reported: 2016-03-11 07:58 CET by Florian Best
Modified: 2016-03-17 14:28 CET (History)
1 user (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Ticket number:
Bug group (optional): External feedback
Max CVSS v3 score:


Note You need to log in before you can comment on or make changes to this bug.
Description Florian Best univentionstaff 2016-03-11 07:58:32 CET
We received the following traceback, 4.1-1 errata128 (Vahr).

I have no clue how that can happen.

Die Ausführung des Kommandos appcenter/query ist fehlgeschlagen:

Traceback (most recent call last):
  File "%PY2.7%/univention/management/console/base.py", line 283, in execute
    function(self, request)
  File "%PY2.7%/univention/management/console/modules/decorators.py", line 320, in _response
    self.finished(request.id, result[0])
  File "%PY2.7%/univention/management/console/base.py", line 401, in finished
  File "%PY2.7%/univention/management/console/base.py", line 405, in result
    self.signal_emit('success', response)
  File "%PY2.7%/notifier/signals.py", line 75, in signal_emit
    self.__signals[ signal ].emit( *args )
  File "%PY2.7%/notifier/signals.py", line 41, in emit
    cb( *args )
  File "%PY2.7%/notifier/__init__.py", line 104, in __call__
    return self._function( *tmp, **self._kwargs )
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 118, in _reply
  File "%PY2.7%/univention/management/console/protocol/modserver.py", line 350, in response
    self.__queue += str(msg)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 337, in __str__
    return Message._formattedMessage(self._id, self._type, self.mimetype, self.command, body, self.arguments)
  File "%PY2.7%/univention/management/console/protocol/message.py", line 130, in _formattedMessage
    data = json.dumps(body)
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 321, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 237, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 311, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 213, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <univention.appcenter.app.AppAttribute object at 0x25d0c10> is not JSON serializable

We should also adapt the __repr__ method of AppAttribute, … during fixing this bug.
Comment 1 Dirk Wiesenthal univentionstaff 2016-03-11 14:27:57 CET
Reported again
Comment 2 Dirk Wiesenthal univentionstaff 2016-03-14 00:40:19 CET
Probably hard to reproduce. It is a race condition: The preinst of univention-appcenter deletes the cache, but some other script rebuilds it _before the new python-univention-appcenter files are extracted_. Thus, newly introduced attributes are missing in the cache. But when the App Center is queried, the new attribute is in the python code, which leads to the error.

The App object never overwrote
  license_description = AppAttribute()

with self.license_description = None, thus the error message.

Now, before reading the cache, app.py, meta.py, index.json.gz are os.stat'ed for last content change. If it is younger than the cache, the cache is deleted and rewritten with fresh data.

Also, the cache file format has been changed from Pickle to JSON to force setting every attribute (at least with None).

This change makes it also possible to compare attributes in the JSON file and the App class. But I did not include this for now, as the os.stat seems enough.

  univention-appcenter 5.0.20-22.133.201603140023
Comment 3 Florian Best univentionstaff 2016-03-14 07:58:05 CET
Reported again, 4.1-1 errata128 (Vahr)
Comment 4 Florian Best univentionstaff 2016-03-15 08:05:58 CET
ucr set 'appcenter/umc/update/always=false'
rm /var/cache/univention-appcenter/.apps.*
comment out website= and vendor= in /usr/share/pyshared/univention/appcenter/app.py
call "univention-app list"
uncomment the attributes in app.py
open the appcenter in UMC
Comment 5 Florian Best univentionstaff 2016-03-15 08:46:13 CET
This isn't really safe against .py/.pyc changes:

>>> from univention.appcenter import AppManager
>>> AppManager._relevant_master_files()
set(['/var/cache/univention-appcenter/.index.json.gz', '/usr/lib/pymodules/python2.7/univention/appcenter/app.py', '/usr/lib/pymodules/python2.7/univention/appcenter/meta.pyc'])

Isn't there a better thing than os.stat()ing - e.g. versionize the API?
A simple touch .apps.de.json .apps.en.json (or modify the ini files remote and remove some attributes + univention-app list) causes the bug to reappear - in theory it could trigger null-pointer exceptions especially for required attributes - the harmless case is that the attributes are just not shown or shown as 'null'.
Comment 6 Florian Best univentionstaff 2016-03-15 09:02:44 CET
Image for the os.stat scenario:
E.g. somewhere in postinst (after the new files app.py are placed with mtime=1 in the file system) a currently still running process (univention-app update, appcenter-UMC with the old code) ends and writes the cache file (mtime=2).
→ Broken - now more harmless - situation again.

The cache file is also nowhere removed anymore.
Comment 7 Florian Best univentionstaff 2016-03-15 10:48:46 CET
Reported again, 4.1-1 errata128 (Vahr)
Comment 8 Dirk Wiesenthal univentionstaff 2016-03-15 11:32:08 CET
The described scenarios are possible, but improbable. But to fix this once and for all, one would need some kind of API version and put it into the cache.

I have implemented a far more simple check in
  univention-appcenter 5.0.20-25.136.201603151120

It compares the attribute _names_ in the cache with those in the App class. If they differ, the cache is rewritten. This can still fail to identify an outdated cache:

- web_interface_proxy_scheme = AppAttribute(default='both', choices=['http', 'https', 'both'])
+ web_interface_proxy_scheme = AppAttribute(default='http', choices=['http', 'https', 'both'])

would not be recognized. I think it is good enough the way it now, though.
Comment 9 Florian Best univentionstaff 2016-03-15 13:30:09 CET
Reported again (twice), 4.1-1 errata128 (Vahr)
Comment 10 Florian Best univentionstaff 2016-03-16 06:47:52 CET
OK: This is nice now.
Comment 11 Philipp Hahn univentionstaff 2016-03-17 14:28:55 CET