First Last Prev Next    This bug is not in your last search results.
Bug 37844 - UMC-Webserver: died during restart
UMC-Webserver: died during restart
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: UMC (Generic)
UCS 4.0
Other Linux
: P5 normal (vote)
: UCS 4.0-2-errata
Assigned To: Florian Best
Sönke Schwardt-Krummrich
https://bitbucket.org/cherrypy/cherry...
:
: 38625 (view as bug list)
Depends on:
Blocks:
  Show dependency tree / graph
 
Reported: 2015-02-19 10:01 CET by Sönke Schwardt-Krummrich
Modified: 2015-08-06 17:55 CEST (History)
4 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:
best: Patch_Available+

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Sönke Schwardt-Krummrich univentionstaff 2015-02-19 10:01:47 CET 
Happened on a UCS@school jenkins instance during restart of the UMC web server.
The UMC web server died during restart!

18.02.15 06:04:59.532  DEBUG_INIT
18.02.15 06:04:59.533  MAIN        ( PROCESS ) : Stopping UMC web server ...
18.02.15 06:04:59.535  MAIN        ( PROCESS ) : The UMC web server is still running. Will wait for 5 seconds
18.02.15 06:05:04.546  MAIN        ( PROCESS ) : The UMC web server is still running. Kill it!
18.02.15 06:05:04.546  MAIN        ( PROCESS ) : Starting UMC web server ...
18.02.15 06:05:04.727  MAIN        ( ERROR   ) : Traceback in run_cherrypy((), {}):
Traceback (most recent call last):
  File "/usr/sbin/univention-management-console-web-server", line 78, in new_f
    return f(*args, **kwargs)
  File "/usr/sbin/univention-management-console-web-server", line 799, in run_cherrypy
    'error_page.default': default_error_page
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpconfig.py", line 156, in update
    reprconf.Config.update(self, config)
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/reprconf.py", line 163, in update
    self._apply(config)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpconfig.py", line 166, in _apply
    reprconf.Config._apply(self, config)
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/reprconf.py", line 175, in _apply
    self.namespaces(config)
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/reprconf.py", line 118, in __call__
    handler(k, v)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpconfig.py", line 255, in _engine_namespace_handler
    engine.autoreload.unsubscribe()
  File "/usr/lib/python2.7/dist-packages/cherrypy/process/plugins.py", line 50, in unsubscribe
    for channel in self.bus.listeners:
RuntimeError: dictionary changed size during iteration
Comment 1 Sönke Schwardt-Krummrich univentionstaff 2015-04-17 09:54:56 CEST 
Happened again in the UCS@school jenkins run (Multiserver-s4-only-master):

19.11.14 08:40:53.791  DEBUG_INIT
10.02.15 07:23:45.612  DEBUG_INIT
10.02.15 07:27:19.398  DEBUG_INIT
10.02.15 07:28:19.996  DEBUG_INIT
10.02.15 07:28:19.997  MAIN        ( PROCESS ) : Stopping UMC web server ...
10.02.15 07:28:19.997  MAIN        ( PROCESS ) : Starting UMC web server ...
15.04.15 05:57:55.117  DEBUG_INIT
15.04.15 06:00:24.896  DEBUG_INIT
15.04.15 06:00:24.898  MAIN        ( PROCESS ) : The UMC web server is still running. Will wait for 5 seconds
15.04.15 06:00:29.905  MAIN        ( PROCESS ) : The UMC web server is still running. Kill it!
15.04.15 06:04:49.626  DEBUG_INIT
15.04.15 06:24:08.542  DEBUG_INIT
15.04.15 06:24:08.543  MAIN        ( PROCESS ) : Stopping UMC web server ...
15.04.15 06:24:08.554  MAIN        ( PROCESS ) : The UMC web server is still running. Will wait for 5 seconds
15.04.15 06:24:13.560  MAIN        ( PROCESS ) : The UMC web server is still running. Kill it!
15.04.15 06:24:13.560  MAIN        ( PROCESS ) : Starting UMC web server ...
15.04.15 06:24:30.416  DEBUG_INIT
15.04.15 06:24:30.420  MAIN        ( PROCESS ) : The UMC web server is still running. Will wait for 5 seconds
15.04.15 06:24:35.435  MAIN        ( PROCESS ) : The UMC web server is still running. Kill it!
15.04.15 06:25:53.954  DEBUG_INIT
15.04.15 06:25:54.091  MAIN        ( ERROR   ) : Traceback in run_cherrypy((), {}):
Traceback (most recent call last):
  File "/usr/sbin/univention-management-console-web-server", line 78, in new_f
    return f(*args, **kwargs)
  File "/usr/sbin/univention-management-console-web-server", line 800, in run_cherrypy
    'error_page.default': default_error_page
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpconfig.py", line 156, in update
    reprconf.Config.update(self, config)
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/reprconf.py", line 163, in update
    self._apply(config)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpconfig.py", line 166, in _apply
    reprconf.Config._apply(self, config)
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/reprconf.py", line 175, in _apply
    self.namespaces(config)
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/reprconf.py", line 118, in __call__
    handler(k, v)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpconfig.py", line 255, in _engine_namespace_handler
    engine.autoreload.unsubscribe()
  File "/usr/lib/python2.7/dist-packages/cherrypy/process/plugins.py", line 50, in unsubscribe
    for channel in self.bus.listeners:
RuntimeError: dictionary changed size during iteration
Comment 2 Sönke Schwardt-Krummrich univentionstaff 2015-05-27 14:08:25 CEST 
Traceback happend again during preparations for the upcoming UCS training on one the 6 machines. The UMC web server was unreachable. Port 8090 was closed. Therefore the login to UMC failed. Additional software has been installed via the UMC app center (DC master; UCS 4.0-2 errata level 194).
Comment 4 Florian Best univentionstaff 2015-06-01 00:11:47 CEST 
*** Bug 38625 has been marked as a duplicate of this bug. ***
Comment 5 Florian Best univentionstaff 2015-06-09 14:53:25 CEST 
QA: check if /usr/lib/python2.6/dist-packages/cherrypy/process/plugins.py line 586 contains "list" and if the web-server still starts.

Package: cherrypy3
Version: 3.2.2-2.15.201506091410
Branch: ucs_4.0-0
Scope: errata4.0-2

YAML: 2015-06-09-cherrypy3.yaml
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2015-06-19 14:37:35 CEST 
(In reply to Florian Best from comment #5)
> QA: check if /usr/lib/python2.6/dist-packages/cherrypy/process/plugins.py
> line 586 contains "list" and if the web-server still starts.

The changed line does not seem to match with the tracebacks. They are both in line 50 of plugins.py (not 586).
Comment 7 Florian Best univentionstaff 2015-07-03 01:32:52 CEST 
I found the real reason for this error.
We are running cherrypy in a thread. In the started cherrypy thread the cherrypy configuration is updated which removes the autoloader plugin from the cherrypy bus. In the main thread we are registering a signal handler plugin at the cherrypy bus. If this happens at the same time the traceback occurs.

Removed cherrypy-YAML, removed cherrypy-patch, rebuild cherrypy.

univention-management-console-frontend (4.1.106-71):
r61686 | Bug #37844: fix race condition which caused failing web-server startups

2015-05-28-univention-management-console-frontend.yaml:
r61687 | YAML Bug #37844
Comment 8 Florian Best univentionstaff 2015-07-07 13:05:28 CEST 
Introduced by Bug #37317 / svn r56826
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2015-07-27 13:48:52 CEST 
OK: Removed cherrypy-YAML
OK: removed cherrypy-patch
OK: rebuild cherrypy

> univention-management-console-frontend (4.1.106-71):
> r61686 | Bug #37844: fix race condition which caused failing web-server
> startups

Was not able to reproduce/trigger the race condition.
OK: code change
OK: functional change (UCR reload via SIGHUP is still functional)
 
> 2015-05-28-univention-management-console-frontend.yaml:
> r61687 | YAML Bug #37844

OK: YAML
Comment 10 Janek Walkenhorst univentionstaff 2015-08-06 17:55:04 CEST 
<http://errata.univention.de/ucs/4.0/266.html>
First Last Prev Next    This bug is not in your last search results.