Description Florian Best 2022-07-06 15:19:52 CEST

Every host executes every monitoring plugin check script and therefore writes metric-data to prometheus via the prometheus node exporter.

The alert expressions (part of the alert objects in LDAP) aren't bound to specific hosts.
E.g. an expression is `univention_cups_running != 1` and not `univention_cups_running{hostname=primary.domain} != 1`.

If such an alert object is not assigned to a certain host (but to any other) the configuration is still written and evaluated.
This then fires an alert because the system which doesn't have e.g. cups installed writes `univention_cups_running = 0` into prometheus.

We should find a mechanism to bound the alerts to certain hosts.
1. extend the expression in the listener by adding "{instance=…}" checks. This requires parsing and understanding the expressions.
Advantage:
* would be correct from the prometheus point of view
Disadvantage:
* a very long expression could lead to performance problems? (e.g. with 100 hosts assigned)
* usually in prometheus this is often done via substring machting
* writing/using a parser/lexer is very complex and error prone

2. disable that the scripts are writing the metric-data to prometheus (this is currently already possible via UCR for each check)

For this we have to define a mapping of alert <> script.
* could be an attribute of the alert object
* could be some local file shipped by the script

Additionally we should add LDAP ACL's for DC/Memberservers which allow to adjust/create the alert objects. This makes update handling more easily, so that joinscript versions don't need to be increased during errata updates.
 
Additionally we have to remove the check if the assigned hostname of an alert is the current hostdn of the server. otherwise prometheus must be installed on the host where the alert is assigned to.