Univention Bugzilla – Bug 40239
Performance optimization for App Center: Pickle Apps
Last modified: 2016-03-15 09:05:17 CET
Apps could be pickled once they are loaded and reread from the pickle file instead of parsing all ini files again and again. Care has to be taken when to delete the pickle files. One file per locale! +++ This bug was initially created as a clone of Bug #39632 +++ Startup performance for the UMC module is mediocre. We should elaborate ways to enhance execution speed. One way might be caching of UCR. Currently, UCR is initialized in a lot of local functions. This way one does not have to think about thread safety, concurrent processes and so on. But it may cost too much time. LDAP lookups may also be reduced, although this has already seen one patch.
Fixed in univention-appcenter 5.0.19-28.86.201512140023
Does pickling really help w.r.t. performance? I am not convinced. IMHO pickling makes the code more complicated too understand and may possibly lead to problems due to outdated pickle files (they would be a cache of the cache of the online App Center data). Instead, I would vote for reading INI files and caching them within the process memory (in case they are read again and again). It would then just need a smart check (e.g., timestamps and file size) in order to determine whether an INI file needs to be parsed again.
(In reply to Alexander Kläser from comment #2) > Does pickling really help w.r.t. performance? I am not convinced. IMHO > pickling makes the code more complicated too understand and may possibly > lead to problems due to outdated pickle files (they would be a cache of the > cache of the online App Center data). # univention-app update --ucs-version 4.0 # getting a lot of files... # rm /var/cache/univention-appcenter/.apps.* # time univention-app get samba4 version Version: 4.1 real 0m3.041s user 0m1.384s sys 0m0.080s # time univention-app get samba4 version Version: 4.1 real 0m1.521s user 0m0.656s sys 0m0.068s > Instead, I would vote for reading INI files and caching them within the > process memory (in case they are read again and again). It would then just > need a smart check (e.g., timestamps and file size) in order to determine > whether an INI file needs to be parsed again. I would not do that. Not only does it not help in what we want (faster startup of App Center, as it would read the ini files each first opening), the code would most probably be more complicated as it is now. Plus we would reimplement some kind of hard disk caching in Python? This does not seem to be a good idea. Plus this patch is more or less just a few lines in one function.
I tried to measure the time spent, as well. Yes, it looks better: -------------------- 8< -------------------- root@ucs-1873:~# apt-cache policy python-univention-appcenter python-univention-appcenter: Installiert: 5.0.19-19.78.201512031352 [...] root@ucs-1873:~# time python -c 'from univention.appcenter import app; print len(app.AppManager.get_all_apps())' 69 real 0m3.304s user 0m1.924s sys 0m0.132s root@ucs-1873:~# time python -c 'from univention.appcenter import app; print len(app.AppManager.get_all_apps())' 69 real 0m1.912s user 0m1.724s sys 0m0.168s root@ucs-1873:~# time python -c 'from univention.appcenter import app; print len(app.AppManager.get_all_apps())' 69 real 0m2.015s user 0m1.820s sys 0m0.168s root@ucs-1873:~# time univention-app get samba4 version Version: 4.3 real 0m1.571s user 0m1.396s sys 0m0.156s root@ucs-1873:~# time univention-app get samba4 version Version: 4.3 real 0m1.580s user 0m1.420s sys 0m0.136s root@ucs-1873:~# time univention-app get samba4 version Version: 4.3 real 0m1.581s user 0m1.424s sys 0m0.136s root@ucs-1873:~# apt-get install python-univention-appcenter [...] root@ucs-1873:~# apt-cache policy python-univention-appcenter python-univention-appcenter: Installiert: 5.0.19-30.88.201512160029 [...] root@ucs-1873:~# time python -c 'from univention.appcenter import app; print len(app.AppManager.get_all_apps())' 69 real 0m1.447s user 0m1.296s sys 0m0.136s root@ucs-1873:~# time python -c 'from univention.appcenter import app; print len(app.AppManager.get_all_apps())' 69 real 0m1.152s user 0m0.996s sys 0m0.136s root@ucs-1873:~# time python -c 'from univention.appcenter import app; print len(app.AppManager.get_all_apps())' 69 real 0m1.141s user 0m1.000s sys 0m0.124s root@ucs-1873:~# time univention-app get samba4 version Version: 4.3 real 0m1.246s user 0m1.100s sys 0m0.120s root@ucs-1873:~# time univention-app get samba4 version Version: 4.3 real 0m1.296s user 0m1.132s sys 0m0.124s root@ucs-1873:~# time univention-app get samba4 version Version: 4.3 real 0m1.259s user 0m1.092s sys 0m0.128s -------------------- 8< --------------------
OK: CLI + UMC is faster OK: cache invalidation OK: package update OK: no conflict with the ini cache (e.g. UMC-module-appcenter preinst). OK~: YAML BTW: > 35 for pkl in $(ls /var/cache/univention-appcenter/.apps.*.pkl); do > 36 » rm "$pkl" > 37 done Could have been: > rm /var/cache/univention-appcenter/.apps.*.pkl
<http://errata.software-univention.de/ucs/4.1/79.html>
(In reply to Alexander Kläser from comment #2) > Does pickling really help w.r.t. performance? I am not convinced. IMHO > pickling makes the code more complicated too understand and may possibly > lead to problems due to outdated pickle files (they would be a cache of the > cache of the online App Center data). The side effects due to outdated pickle files are Bug #40882, Bug #40875.