Univention Bugzilla – Bug 50411
umc appcenter module memory consumption too high?
Last modified: 2019-11-05 18:03:17 CET
see alos Bug #48554 Seems that the appcenter umc module consumes a lot of mem, just opened the Appcenter in UMC and 30% memory "gone". -> ps root 26009 1.8 35.2 955320 359768 ? S 01:18 0:07 /usr/bin/python2.7 /usr/sbin/univention-management-console-module -m appcenter -s /var/run/univention-management-console/14786-1571786287842.socket -d 2 -l de_DE.UTF-8 root 26010 0.2 5.6 507452 57436 ? S 01:18 0:00 /usr/bin/python2.7 /usr/sbin/univention-management-console-module -m join -s /var/run/univention-management-console/14786-1571786287884.socket -d 2 -l de_DE.UTF-8 root 26011 0.2 7.8 540020 80256 ? S 01:18 0:01 /usr/bin/python2.7 /usr/sbin/univention-management-console-module -m updater -s /var/run/univention-management-console/14786-1571786287911.socket -d 2 -l de_DE.UTF-8 root 26028 0.2 7.3 610468 74744 ? S 01:18 0:01 /usr/bin/python2.7 /usr/sbin/univention-management-console-module -m udm -s /var/run/univention-management-console/14786-1571786290549.socket -d 2 -l de_DE.UTF-8 -> free -m total used free shared buff/cache available Mem: 996 615 66 10 314 232 Swap: 2047 504 1543 This is particularly nasty in our jenkins tests, as they start multiple umc appcenter processes (before the umc module timeout hits). Can this be optimized?
ucs-test: 91ed5e40dfd6a9adfc243baf489693f4aa46c346 added some umc restarts in 20_appcenter as a workaround
memory profile for umc init() and query() with memory_profiler ---- from memory_profiler import profile f=open('/tmp/log.txt','w+') ... @profile(stream=f) def init(self): ---- Filename: /usr/lib/python2.7/dist-packages/univention/management/console/modules/appcenter/__init__.py Line # Mem usage Increment Line Contents ================================================ 134 60.5 MiB 0.0 MiB @profile(stream=f) 135 def init(self): 136 60.5 MiB 0.0 MiB os.umask(0o022) # umc umask is too restrictive for app center as it creates a lot of files in docker containers 137 60.5 MiB 0.0 MiB self.ucr = ucr_instance() 138 139 60.5 MiB 0.0 MiB self.update_applications_done = False 140 60.5 MiB 0.0 MiB util.install_opener(self.ucr) 141 60.5 MiB 0.0 MiB self._is_working = False 142 60.5 MiB 0.0 MiB self._remote_progress = {} 143 144 60.5 MiB 0.0 MiB try: 145 60.5 MiB 0.0 MiB self.package_manager = PackageManager( 146 60.5 MiB 0.0 MiB info_handler=MODULE.process, 147 60.5 MiB 0.0 MiB step_handler=None, 148 60.5 MiB 0.0 MiB error_handler=MODULE.warn, 149 135.5 MiB 75.0 MiB lock=False, 150 ) 151 except SystemError as exc: 152 MODULE.error(str(exc)) 153 raise umcm.UMC_Error(str(exc), status=500) 154 135.5 MiB 0.0 MiB self.package_manager.set_finished() # currently not working. accepting new tasks 155 135.5 MiB 0.0 MiB get_package_manager._package_manager = self.package_manager 156 157 # build cache 158 146.0 MiB 10.5 MiB _update_modules() 159 269.1 MiB 123.2 MiB get_action('list').get_apps() 160 161 # not initialize here: error prone due to network errors and also kinda slow 162 269.1 MiB 0.0 MiB self._uu = None 163 269.1 MiB 0.0 MiB self._cm = None 164 165 # in order to set the correct locale 166 269.1 MiB 0.0 MiB locale.setlocale(locale.LC_ALL, str(self.locale)) 167 168 269.1 MiB 0.0 MiB try: 169 269.1 MiB 0.0 MiB log_to_logfile() 170 except IOError: 171 pass 172 173 # connect univention.appcenter.log to the progress-method 174 269.1 MiB 0.0 MiB handler = ProgressInfoHandler(self.package_manager) 175 269.1 MiB 0.0 MiB handler.setLevel(logging.INFO) 176 269.1 MiB 0.0 MiB get_base_logger().addHandler(handler) 177 178 269.1 MiB 0.0 MiB percentage = ProgressPercentageHandler(self.package_manager) 179 269.1 MiB 0.0 MiB percentage.setLevel(logging.DEBUG) 180 269.1 MiB 0.0 MiB get_base_logger().getChild('actions.install.progress').addHandler(percentage) 181 269.1 MiB 0.0 MiB get_base_logger().getChild('actions.upgrade.progress').addHandler(percentage) 182 269.1 MiB 0.0 MiB get_base_logger().getChild('actions.remove.progress').addHandler(percentage) Filename: /usr/lib/python2.7/dist-packages/univention/management/console/modules/appcenter/__init__.py Line # Mem usage Increment Line Contents ================================================ 254 269.1 MiB 0.0 MiB @simple_response 255 @profile(stream=f) 256 def query(self, quick=False): 257 269.1 MiB 0.0 MiB if not quick: 258 269.3 MiB 0.2 MiB self.update_applications() 259 269.7 MiB 0.4 MiB self.ucr.load() 260 269.2 MiB -0.5 MiB reload_package_manager This is 270 MiB just for the list of available apps.