Bug 50411 - umc appcenter module memory consumption too high?
umc appcenter module memory consumption too high?
Status: NEW
Product: UCS
Classification: Unclassified
Component: App Center
UCS 4.4
Other Linux
: P5 normal (vote)
: ---
Assigned To: App Center maintainers
App Center maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2019-10-24 14:15 CEST by Felix Botner
Modified: 2019-11-05 18:03 CET (History)
1 user (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional): UCS Performance
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Felix Botner univentionstaff 2019-10-24 14:15:49 CEST
see alos Bug #48554

Seems that the appcenter umc module consumes a lot of mem, just opened the Appcenter in UMC and 30% memory "gone".

-> ps

root     26009  1.8 35.2 955320 359768 ?       S    01:18   0:07 /usr/bin/python2.7 /usr/sbin/univention-management-console-module -m appcenter -s /var/run/univention-management-console/14786-1571786287842.socket -d 2 -l de_DE.UTF-8
root     26010  0.2  5.6 507452 57436 ?        S    01:18   0:00 /usr/bin/python2.7 /usr/sbin/univention-management-console-module -m join -s /var/run/univention-management-console/14786-1571786287884.socket -d 2 -l de_DE.UTF-8
root     26011  0.2  7.8 540020 80256 ?        S    01:18   0:01 /usr/bin/python2.7 /usr/sbin/univention-management-console-module -m updater -s /var/run/univention-management-console/14786-1571786287911.socket -d 2 -l de_DE.UTF-8
root     26028  0.2  7.3 610468 74744 ?        S    01:18   0:01 /usr/bin/python2.7 /usr/sbin/univention-management-console-module -m udm -s /var/run/univention-management-console/14786-1571786290549.socket -d 2 -l de_DE.UTF-8

-> free -m
              total        used        free      shared  buff/cache   available
Mem:            996         615          66          10         314         232
Swap:          2047         504        1543

This is particularly nasty in our jenkins tests, as they start multiple umc appcenter processes (before the umc module timeout hits).

Can this be optimized?
Comment 1 Felix Botner univentionstaff 2019-10-29 11:50:31 CET
ucs-test: 91ed5e40dfd6a9adfc243baf489693f4aa46c346
added some umc restarts in 20_appcenter as a workaround
Comment 2 Felix Botner univentionstaff 2019-11-05 18:03:17 CET
memory profile for umc init() and query() with memory_profiler

----
from memory_profiler import profile
f=open('/tmp/log.txt','w+')
...

    @profile(stream=f)
    def init(self):
----




Filename: /usr/lib/python2.7/dist-packages/univention/management/console/modules/appcenter/__init__.py

Line #    Mem usage    Increment   Line Contents
================================================
   134     60.5 MiB      0.0 MiB   	@profile(stream=f)
   135                             	def init(self):
   136     60.5 MiB      0.0 MiB   		os.umask(0o022)  # umc umask is too restrictive for app center as it creates a lot of files in docker containers
   137     60.5 MiB      0.0 MiB   		self.ucr = ucr_instance()
   138                             
   139     60.5 MiB      0.0 MiB   		self.update_applications_done = False
   140     60.5 MiB      0.0 MiB   		util.install_opener(self.ucr)
   141     60.5 MiB      0.0 MiB   		self._is_working = False
   142     60.5 MiB      0.0 MiB   		self._remote_progress = {}
   143                             
   144     60.5 MiB      0.0 MiB   		try:
   145     60.5 MiB      0.0 MiB   			self.package_manager = PackageManager(
   146     60.5 MiB      0.0 MiB   				info_handler=MODULE.process,
   147     60.5 MiB      0.0 MiB   				step_handler=None,
   148     60.5 MiB      0.0 MiB   				error_handler=MODULE.warn,
   149    135.5 MiB     75.0 MiB   				lock=False,
   150                             			)
   151                             		except SystemError as exc:
   152                             			MODULE.error(str(exc))
   153                             			raise umcm.UMC_Error(str(exc), status=500)
   154    135.5 MiB      0.0 MiB   		self.package_manager.set_finished()  # currently not working. accepting new tasks
   155    135.5 MiB      0.0 MiB   		get_package_manager._package_manager = self.package_manager
   156                             
   157                             		# build cache
   158    146.0 MiB     10.5 MiB   		_update_modules()
   159    269.1 MiB    123.2 MiB   		get_action('list').get_apps()
   160                             
   161                             		# not initialize here: error prone due to network errors and also kinda slow
   162    269.1 MiB      0.0 MiB   		self._uu = None
   163    269.1 MiB      0.0 MiB   		self._cm = None
   164                             
   165                             		# in order to set the correct locale
   166    269.1 MiB      0.0 MiB   		locale.setlocale(locale.LC_ALL, str(self.locale))
   167                             
   168    269.1 MiB      0.0 MiB   		try:
   169    269.1 MiB      0.0 MiB   			log_to_logfile()
   170                             		except IOError:
   171                             			pass
   172                             
   173                             		# connect univention.appcenter.log to the progress-method
   174    269.1 MiB      0.0 MiB   		handler = ProgressInfoHandler(self.package_manager)
   175    269.1 MiB      0.0 MiB   		handler.setLevel(logging.INFO)
   176    269.1 MiB      0.0 MiB   		get_base_logger().addHandler(handler)
   177                             
   178    269.1 MiB      0.0 MiB   		percentage = ProgressPercentageHandler(self.package_manager)
   179    269.1 MiB      0.0 MiB   		percentage.setLevel(logging.DEBUG)
   180    269.1 MiB      0.0 MiB   		get_base_logger().getChild('actions.install.progress').addHandler(percentage)
   181    269.1 MiB      0.0 MiB   		get_base_logger().getChild('actions.upgrade.progress').addHandler(percentage)
   182    269.1 MiB      0.0 MiB   		get_base_logger().getChild('actions.remove.progress').addHandler(percentage)


Filename: /usr/lib/python2.7/dist-packages/univention/management/console/modules/appcenter/__init__.py

Line #    Mem usage    Increment   Line Contents
================================================
   254    269.1 MiB      0.0 MiB   	@simple_response
   255                             	@profile(stream=f)
   256                             	def query(self, quick=False):
   257    269.1 MiB      0.0 MiB   		if not quick:
   258    269.3 MiB      0.2 MiB   			self.update_applications()
   259    269.7 MiB      0.4 MiB   		self.ucr.load()
   260    269.2 MiB     -0.5 MiB   		reload_package_manager


This is 270 MiB just for the list of available apps.