Bug 46901 - Make mysql startup timeout configureable via UCR
Make mysql startup timeout configureable via UCR
Status: CLOSED FIXED
Product: UCS
Classification: Unclassified
Component: MySQL
UCS 4.4
Other Linux
: P5 normal (vote)
: UCS 4.4-0-errata
Assigned To: Julia Bremer
Erik Damrose
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-04-27 12:19 CEST by Erik Damrose
Modified: 2019-03-20 12:29 CET (History)
3 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 1: Will affect a very few installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.086
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Erik Damrose univentionstaff 2018-04-27 12:19:45 CEST
MySQL / MariaDB can take a long time to startup on boot. It is using an initscript to determine if the daemon is running, which by default checks for 30 seconds. Otherwise, the systemd running type is registered as failed. When (re-)starting the daemon, the init script picks up any running mysqld process and marks the systemd unit as running - but this daemon does may have any configuration changes applied that were done since boot.

In a first step we can make the initscript timout configurable via UCR by making /etc/default/mysql a UCR template to configure MYSQLD_STARTUP_TIMEOUT

A concrete problem scenario was, that mysql was already installed, but failed during boot. Upon a docker app installation, univention-mysql was installed to configure the mysql bind address to 0.0.0.0, but the mysql restart did not pick up this configuration change (see above). The docker app could then not connect to the database.
Comment 1 Sven Anders 2018-06-13 09:36:56 CEST
Happens on a second system. The system was booted during the night, when a backup on VMware "eats" hard disk IO performance.
Comment 2 Julia Bremer univentionstaff 2018-10-30 17:34:29 CET
Ok, MYSQLD_STARTUP_TIMEOUT is evaluated in /xetc/init.d/mysql but there is also a native systemd service file:

root@jb5700mc:~# ls -ld /etc/systemd/system/mysql.service \
                        /lib/systemd/system/mariadb.service
lrwxrwxrwx 1 root root   35 Aug  7 19:26 /etc/systemd/system/mysql.service -> /lib/systemd/system/mariadb.service
-rw-r--r-- 1 root root 4505 Aug 10  2017 /lib/systemd/system/mariadb.service

So, I'm not sure yet, that and adjustment of the TIMEOUT variable will change anything for the systemd behaviour.


The second question I have is about how we should introduce the UCR adjustable value for MYSQLD_STARTUP_TIMEOUT into the defaults file. Since univention-mysql was not installed in your case, we would have to patch the upstream package directly?
Comment 3 Julia Bremer univentionstaff 2019-03-13 17:00:14 CET
Successful build
Package: univention-mariadb
Version: 2.0.0-4A~4.4.0.201903131631
Branch: ucs_4.4-0
Scope: errata4.4-0

b1aaf6707f Bug #46901: Version bump
0edc1d9ab1 Bug #46901: Fix typo
38559867ea Bug #46901: mariadb startup timeout now configurable via ucr
a367fed1bd Bug #46901: YAML

Since MariaDB uses systemd to start the service, I created a new config file under /lib/systemd/system/mariadb.service.d
You can now set the systemd option TimeoutStartSec via the ucr-variable 
mariadb/startup/timeout.
Comment 4 Erik Damrose univentionstaff 2019-03-18 12:50:43 CET
The implementation looks very good, also, thanks for the detailed documentation.
OK: Systemd service file
OK: UCRv mariadb/startup/timeout + documentation
OK: Default value of 90
OK: Test of timeout (i adjusted /etc/mysql/debian-start for my test)
OK: Yaml

Reopen: One small issue: The systemd config is not updated on package installation, the following warning is shown:

Warning: mariadb.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Apparently one has to reload the service manually in the postinst. In a short test i tried adding the systemd debhelper to debian/rules, but it did not work. It seems the mysql init script prevents the debhelper from being added. Even the mariadb-server postinst has a note before a manual reload: "# Modified dh_systemd_start snippet that's not added automatically due /etc/init.d/mysql"

So we should add the following to univention-mariadb.postinst, before the service restart command:
systemctl --system daemon-reload >/dev/null
Comment 5 Julia Bremer univentionstaff 2019-03-19 15:58:35 CET
Successful build
Package: univention-mariadb
Version: 2.0.0-5A~4.4.0.201903191553
Branch: ucs_4.4-0
Scope: errata4.4-0

2cf00085d7 Bug #46901: Yaml
6865da53cf Bug #46901: Make system daemon-reload in postinst of mariadb

I added the line you suggested to the postinst.
Comment 6 Erik Damrose univentionstaff 2019-03-19 17:19:52 CET
Thanks, works great. I slightly adapted the yaml file, one line was longer than the allowed length, commit 046cf14

Verified
Comment 7 Arvid Requate univentionstaff 2019-03-20 12:29:25 CET
<http://errata.software-univention.de/ucs/4.4/7.html>