Univention Bugzilla – Bug 42658
ucs-test-ucsschool: UMC tests fail with HTTP 511
Last modified: 2019-02-05 21:43:46 CET
A lot of UMC tests fail on both master and slave systems with: httplib.HTTPException: 500 on slave2032 (computerroom/room/acquire): {"status": 511, "message": "Connection to module process failed"} http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.0%20to%204.1%20Singleserver/SambaVersion=s4/207/testReport/junit/90_ucsschool/21_computerroom_module_base_checks/test/ http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.0%20to%204.1%20Singleserver/SambaVersion=s4/207/testReport/junit/90_ucsschool/70_users_module/test/ http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.0%20to%204.1%20Singleserver/SambaVersion=s4/207/testReport/junit/90_ucsschool/73_CSV_import_module/test/ http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.1%20(R2)%20Multiserver/SambaVersion=s4-school-only/268/testReport/junit/90_ucsschool/25_room_management_module/test/ http://jenkins.knut.univention.de:8080/job/UCSschool%204.1/job/UCSschool%204.1%20(R2)%20Multiserver/SambaVersion=s4/268/testReport/junit/90_ucsschool/21_computerroom_module_base_checks/test/
We need the log files /var/log/univention/management-console*.log
511 means: A) a unknown exception occurred during mod.connect() B) the connection could not be established after 200 attempts (every 50 milliseconds a attempt is done which sums up to 10 seconds). I analyzed one of the failing tests: UMC-Server creates the process: > 06.11.16 22:34:08.562 MAIN ( INFO ) : Starting new module process and passing new request to module internetrules: 147848964828673-56 UMC-Module starts: > 06.11.16 22:34:09.003 DEBUG_INIT 06.11.16 22:34:11.877 MAIN ( INFO ) : No connection to module process yet 06.11.16 22:34:15.118 MAIN ( INFO ) : No connection to module process yet 06.11.16 22:34:17.622 MAIN ( INFO ) : No connection to module process yet 06.11.16 22:34:20.118 MAIN ( INFO ) : No connection to module process yet 06.11.16 22:34:20.168 MAIN ( INFO ) : Connection to module internetrules process failed → this is the indication of case B The last message in the logfile of the module is: 06.11.16 22:34:12.675 ADMIN ( INFO ) : admin.syntax.import_hook_files: importing "/usr/lib/pymodules/python2.7/univention/admin/hooks.d/h3fybs16kx.py" So what happened between 22:34:12 and 22:34:20 ? That are 8 seconds without anything. Unfortunately there is no logging in UMC between import and initialization of the base class. But there is a log message before socket initialization and the constructor of the module is empty. So this is caused by the import. We are doing at least twice "udm_modules.update()" in UCS@school modules at import time (schoolldap.py and models/__init__.py). In the logfiles are a lot of uncleaned hooks and syntax class imports from other test scripts. The other test scripts should clean up correctly.
On a regular system there are 227 univention/ucsschool imports. I don't see any blocking import. The dead UDM hooks/syntaxes/modules are caused by Bug #42862.
Created attachment 8214 [details] patch The tests aren't failing anymore because I monkeypatched the time limit. We should nevertheless enhance the UCS@school lib like in the patch. If this doesn't fix it completely we could increase our IO EC2 capacity. ucs-ec2-tools (2.0.0-29): r74237 | Bug #42658: add monkeypatch for UCS@school tests
Like in bug 40727 comment 3 I reduced the memory usage of the test by reusing the umc connection. Instead of 15 modules only one is opened during the test. That should hopefully stabilize the test. [4.3 ab80c5919] Bug #42658: reuse umc connection to save memory
[4.3 25ec9e76f] Bug #42658: reuse umc connection to save memory (2)
[4.3 f4dcb7b4f] Bug #42658: use master umc connection (10_assign_internet_rules_check)
This issue has been filled against UCS@school 4.1 (R2). The maintenance with bug and security fixes for UCS@school 4.1 (R2) has ended on 5th of April 2018. Customers still on UCS 4.1 are encouraged to update to UCS 4.3 (or later). Please contact your partner or Univention for any questions. If this issue still occurs in newer UCS versions, please use "Clone this bug" or simply reopen the issue. In this case please provide detailed information on how this issue is affecting you.