Univention Bugzilla – Bug 54452
Race condition in App Center cache invalidation+rebuild can cause App Center listener converter services to fail
Last modified: 2022-04-28 10:59:06 CEST
This still applies to 5.0-1 and has become much worse. +++ This bug was initially created as a clone of Bug #51986 +++ Due to a race condition during the rebuild of the App Center cache an app listener converter service may stop running, affected apps will then no longer process user or group changes. When univention-app update is running, the app center cache is invalidated, new metafiles are downloaded and extracted. The first usage of appcenter python library functions finds no cache and rebuilds it, e.g. from univention.appcenter.app_cache import Apps Apps().find(<appid>) The race condition can happen when the univention-app update process is still unpacking the downloaded metafiles while another process trigger the cache rebuild. The rebuild is done by taking the app ini files in certain directories and building the cache from them. As the cache is then present, there is no mechanism to detect that not all files have been unpacked and the cache has to be rebuild. That can lead to the case that an app is missing in the cache. The App Center listener converter tries to get the App object from the cache upon starting, if that fails, no processing will happen. The check is done in management/univention-appcenter/scripts/univention-appcenter-listener-converter:182 ... app = Apps().find(args.app) if not app: logger.critical('App not found') sys.exit(1) Symptoms: /var/log/univention/listener_modules/<appid>.log shows 2020-09-10 10:01:48 CRITICAL App not found # service univention-appcenter-listener-converter@<appid> status will show Sep 09 16:40:21 ucs systemd[1]: univention-appcenter-listener-converter@<appid>.service: Main process exited, code=exited, status=1/FAILURE Sep 09 16:40:21 ucs systemd[1]: univention-appcenter-listener-converter@<appid>.service: Unit entered failed state. Sep 09 16:40:21 ucs-ma systemd[1]: univention-appcenter-listener-converter@ucs-to-school-transformer.service: Failed with result 'exit-code'. Another univention-app update will usually not invalidate the cache immediately, because the cache is only invalidated when the metadata files changed on the app center server. The chance for the problem to occur rises the more apps are installed that use a listener converter, as each service gets triggered every 5 seconds. The affected customer system has 3 apps with converters installed. Workaround: remove cache files to force a rebuild with the next access to the cache. (Check if the test app center is used and adapt the path below accordingly) rm /var/cache/univention-appcenter/appcenter.software-univention.de/4.4/.apps.*json
- [x] Code https://git.knut.univention.de/univention/ucs/-/merge_requests/275 - [ ] Test - [ ] Review - [ ] Import and build
Minor fixes were done in the feature branch. After review the code was merged and build for errata5.0-1: 2405d13e merge a64dfe85 changelog 6dd7423d yaml univention-appcenter 9.0.2-72A~5.0.0.202202141913
We will have to wait for test results. But my manual tests were successful. The all.tar.gz is now downloaded as .tmp.tar and the cache files are extracted before the temporary file is renamed to its "official" name .all.tar. This should greatly reduce any race condition. This works as expected and I did not find an error in the code nor in many, many, cache updates.
The test 80_docker.59_app_center_signature currently fails when doing an `app update`, because the file downloaded in "def _download_apps" is now called .tmp.tar while the corresponding gpg file is still saved as all.tar.gpg. The signature verification therefore fails. http://jenkins.knut.univention.de:8080/job/UCS-5.0/job/UCS-5.0-1/job/AutotestJoin/lastCompletedBuild/SambaVersion=no-samba,Systemrolle=master-part-II/testReport/80_docker/59_app_center_signature/master090/
[5.0-1] 645e74a0df style[AppC]: Re-indent .../univention-appcenter/python/appcenter/actions/update.py | 68 ++++++++++++++--------------- 1 file changed, 34 insertions(+), 34 deletions(-) [5.0-1] ea361de870 fix[AppC]: Verify .tmp.tar with .all.tar.gpg .../univention-appcenter/python/appcenter/actions/update.py | 36 ++++++++++++++++------------- 1 file changed, 20 insertions(+), 16 deletions(-) [5.0-1] 82e93d845c feat[AppC]: Specify signature file name management/univention-appcenter/python/appcenter/utils.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) [5.0-1] 4aa8ea379d refactor[AppC]: inverted download logic management/univention-appcenter/python/appcenter/actions/update.py | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) [5.0-1] 4427054233 doc[AppC]: Fix PEP-484 type annotations management/univention-appcenter/python/appcenter/actions/update.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) Package: univention-appcenter Version: 9.0.2-72A~5.0.0.202202151205 Branch: ucs_5.0-0 Scope: errata5.0-1 [5.0-1] b4b0eb8a7a Bug #53523: univention-appcenter 9.0.2-72A~5.0.0.202202151205 doc/errata/staging/univention-appcenter.yaml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) OK: manual tests OK: ucs-test -E dangerous -c -F raw -s docker -s appcenter
[5.0-1] 79ad90dd6d fix[AppC]: Do not delete cache files management/univention-appcenter/python/appcenter/actions/update.py | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) Package: univention-appcenter Version: 9.0.2-72A~5.0.0.202202151637 Branch: ucs_5.0-0 Scope: errata5.0-1 [5.0-1] 5e7d2fb6f0 Bug #53523: univention-appcenter 9.0.2-72A~5.0.0.202202151637 doc/errata/staging/univention-appcenter.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[5.0-1] 5ac57aa763 fix[AppC]: Write app.json atomically management/univention-appcenter/python/appcenter/app_cache.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) Package: univention-appcenter Version: 9.0.2-72A~5.0.0.202202151705 Branch: ucs_5.0-0 Scope: errata5.0-1 [5.0-1] 5e7d2fb6f0 Bug #53523: univention-appcenter 9.0.2-72A~5.0.0.202202151637 doc/errata/staging/univention-appcenter.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
YAML: OK Manual Tests: OK App Center and Test App Center: OK UMC: OK Nightly Tests: OK
<https://errata.software-univention.de/#/?erratum=5.0x224>
Created attachment 10917 [details] proposed pytest Here is the proposed pytest module. If it is conceptually OK - I would add one more, similar to this one but in which half of the reader threads will start first, then writer and then remaining half of the readers