Bug 54640 - UCS@school ID connector fails to start if old pid-file exists
Summary: UCS@school ID connector fails to start if old pid-file exists
Status: CLOSED FIXED
Alias: None
Product: UCS@school
Classification: Unclassified
Component: ucsschool-id-connector
Version: UCS@school 5.0
Hardware: Other Linux
: P5 normal
Target Milestone: ---
Assignee: Ole Schwiegert
QA Contact: Jürn Brodersen
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-04-05 16:20 CEST by Dirk Ahrnke
Modified: 2024-06-13 17:41 CEST (History)
8 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 5: Blocking further progress on the daily work
User Pain: 0.429
Enterprise Customer affected?:
School Customer affected?: Yes
ISV affected?:
Waiting Support: Yes
Flags outvoted (downgraded) after PO Review:
Ticket number: 2023040421000519, 2023100421000347, 2024050321000197
Bug group (optional):
Customer ID: 11818
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Ahrnke univentionstaff 2022-04-05 16:20:40 CEST
This init-script assumes that old pid-files are removed and fails to start once a remnant exists. This may be tha case on unclean shutdown.

/etc/init.d # ./ucsschool-id-connector start
 * Starting UCS@school ID Connector ...
Traceback (most recent call last):
  File "/ucsschool-id-connector/src/queue_management", line 170, in <module>
    _start()
  File "/ucsschool-id-connector/src/queue_management", line 151, in _start
    service.start()
  File "/usr/lib/python3.8/site-packages/service/__init__.py", line 461, in start
    raise ValueError('Daemon is already running at PID %d.' % pid)
ValueError: Daemon is already running at PID 146.                                                                                                                                                                                                       [ !! ]
 * ERROR: ucsschool-id-connector failed to start
/etc/init.d # cat /tmp/IDConnectorService.pid
146
/etc/init.d # 
/etc/init.d # pgrep IDConnectorService
/etc/init.d # rm /tmp/IDConnectorService.pid 
/etc/init.d # ./ucsschool-id-connector start
 * Starting UCS@school ID Connector ...
Started IDConnectorService daemon.                                                                                                                                                                                                                      [ ok ]
/etc/init.d # ./ucsschool-id-connector status
 * status: started
Comment 1 Ole Schwiegert univentionstaff 2022-04-06 07:04:05 CEST
This problem should be fixed, if we introduce s6 as the new init system for our docker containers as can be seen here: https://git.knut.univention.de/univention/components/ucsschool-apis/-/blob/oschwieg/s6/Dockerfile
Comment 4 Daniel Tröder univentionstaff 2022-10-05 12:18:07 CEST
1. When does this happen? What is the scenario here?

2. The official interface is "univention-app".
   What happens if you use "univention-app restart ucsschool-id-connector"?
Comment 5 Jannik Ahlers univentionstaff 2022-10-05 12:32:08 CEST
1. We didn't notice a specific cause or trigger for this. We did have some major problems with the listener/notifier before this was noticed.

2. univention-app restart did not show any errors
'univention-app shell ucsschool-id-connector service ucsschool-id-connector status' returned that the service was stopped.
'univention-app shell ucsschool-id-connector service ucsschool-id-connector start'
returned the error message as Dirk posted above:

 * Starting UCS@school ID Connector ...
Traceback (most recent call last):
  File "/ucsschool-id-connector/src/queue_management", line 170, in <module>
    _start()
  File "/ucsschool-id-connector/src/queue_management", line 151, in _start
    service.start()
  File "/usr/lib/python3.8/site-packages/service/__init__.py", line 461, in start
    raise ValueError('Daemon is already running at PID %d.' % pid)
ValueError: Daemon is already running at PID 286.                                                                                                                                                                                                       [ !! ]
 * ERROR: ucsschool-id-connector failed to start
Comment 6 Jannik Ahlers univentionstaff 2022-10-05 12:32:40 CEST
(forgot to mention, the listener/notifier issues were resolved)
Comment 8 Daniel Tröder univentionstaff 2022-10-05 15:02:59 CEST
This is not a bug. You are using a not-supported way for restarting a service - which will change in the future without notice! As Ole wrote above: we will replace the init system in the Docker container with something that doesn't even have init scripts.

Please use "univention-app restart ucsschool-id-connector".
Comment 9 Jannik Ahlers univentionstaff 2022-10-05 17:03:07 CEST
The issue did not arise because we stopped the service in an unsupported way.
We don't know why the connector stopped working, but it did not happen because of anything we did. It just stopped working, without any error messages.

A resync (as described in the documentation) threw the same 'Daemon is already running' error message as the manual start command, but 'univention-app shell ucsschool-id-connector service ucsschool-id-connector status' returned that the service wasn't running. Only after that did I try to manually start the service.
Comment 11 Daniel Tröder univentionstaff 2023-04-06 07:47:54 CEST
This resets the state (basically deleting the pid file):

/etc/init.d/ucsschool-id-connector zap

After that a start should work. So the "to be sure it's running" commands are:

/etc/init.d/ucsschool-id-connector stop
/etc/init.d/ucsschool-id-connector zap
/etc/init.d/ucsschool-id-connector start
Comment 13 Mirac Erdemiroglu univentionstaff 2023-10-06 12:08:16 CEST
customer affected Ticket#2023100421000347
UCS: 5.0-5 errata821
ucsschool-id-connector=2.2.8 4.3/
ucsschool=5.0 v4
Comment 14 Dirk Ahrnke univentionstaff 2024-02-16 11:45:58 CET
seen again at customer 44145 today. 
Only visible by running "univention-app logs ucsschool-id-connector"
Comment 16 Mirac Erdemiroglu univentionstaff 2024-05-03 17:23:04 CEST
Knowledge Base Article updated:
https://help.univention.com/t/problem-ucs-school-id-connector-fails-to-start-if-old-pid-file-exists/21448
Comment 17 Mirac Erdemiroglu univentionstaff 2024-05-08 16:36:51 CEST
Change the flag, because Ticket 2024050321000197 runs into this issue 2 second time last weekend.

Idea from the customer:

Perhaps the app should be adapted so that the PID file is stored in a tempfs in the future, as with other services that store their PID file under /run, which is empty after a restart.
Comment 19 Johannes Königer univentionstaff 2024-06-13 17:41:17 CEST
Bugfix is released with version 3.0.0.