Univention Bugzilla – Bug 53909
System diagnostic 02_certificate_check.py fails with Let's Encrypt installed
Last modified: 2023-03-18 15:56:23 CET
When Let's Encrypt is used and a new certificate is used by Apache and other services, 02_certificate_check.py fails, because it explicitly tests with the ucsCA/CAcert.pem. We may want to skip this test if Apache's certificate is not signed by our CA. Alternatively or additionally, we could warn the user if there is something different in /usr/local/share/ca-certificates/ than just our CA.
Maybe use /etc/ssl/certs/ca-certificates.crt for checking, as it both contains … - the UCS-CA certificate from /usr/local/share/ca-certificates/ucsCA.crt - all public CA certificates as considered trusted the admin This is what `curl`, `wget`, `firefox`, Python, … uses by default on Debian.
We already tried using the ca-certificates.crt, it does not work. Note: signed_chain.crt used in the following examples is obtained from letsencrypt I see two issues: 1) signed_chain.crt contains the 3 cert chain in the order [host] -> [intermediate] -> [ISRG X1]. openssl verify tries to verify them in-order, and fails at the host cert: $ openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt signed_chain.crt CN = oidc.dev-univention.de error 20 at 0 depth lookup: unable to get local issuer certificate error signed_chain.crt: verification failed 2) When trying to manually reverse the order of certs, one finds that the [ISRG X1] used is the cross-signed one by DST Root CA X3 - which expired on Sep. 30th 2021. openssl verify checks X3 as well, and fails, because X3 is expired. $ openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt signed_chain.reversed.crt O = Digital Signature Trust Co., CN = DST Root CA X3 error 10 at 1 depth lookup: certificate has expired error signed_chain.reversed.crt: verification failed Clients using openssl libs do not have problems when presented with the same cross-signing cert: $ openssl s_client -connect oidc.dev-univention.de:443 CONNECTED(00000005) depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = oidc.dev-univention.de verify return:1 --- Certificate chain 0 s:CN = oidc.dev-univention.de i:C = US, O = Let's Encrypt, CN = R3 1 s:C = US, O = Let's Encrypt, CN = R3 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1 i:O = Digital Signature Trust Co., CN = DST Root CA X3 Note: On all systems used, the ISRG Root X1 cert is present in the local cert store. Note 2: When reversing the order of certs, and removing the cross-signed X1 one, so the cert order is [intermediate] -> [host], openssl verify succeeds.
A (host) certificate is valid if you a full path from that certificate to any trusted anchor can be built, which can include optional intermediate CAs as long as their certificate is also valid. For validity it is not enough for the host certificate to be signed by the intermediate CA only, but the intermediate CA certificate itself must be validated by recursively checking them until a trusted CA is reached. Please be aware that the root CA certificates MUST by definition be self-signed for `verify` to succeed. Otherwise you have to use the option "-partial_chain" if you want to abort early on other explicitly specified trusted intermediate CAs. Looking at /etc/letsencrypt/live/$FQHN/ there are 3 files of interest containing (concatenated) certificates: - cert.pem: 0. subject=CN = $FQHN - chain.pem: 0. subject=C = US, O = Let's Encrypt, CN = R3 1. subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1 (*) - fullchain.pem: 0. subject=CN = $FQHN 1. subject=C = US, O = Let's Encrypt, CN = R3 2. subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1 (*) You have to provide 3 things to `openssl verify`: 1. `-CAfile` The list of *trusted* anchors, e.g. /etc/ssl/certs/ca-certificates.crt 2. `-untrusted` A list of (potentially) initially *untrusted* (intermediate) candidate certificates, which may (optionally) be used to fill the links from a trusted root CA to the certificate to be checked 3. the (host) certificate you want to verify # openssl verify -show_chain -trusted /etc/ssl/certs/ca-certificates.crt -untrusted chain.pem cert.pem # openssl verify -show_chain -trusted /etc/ssl/certs/ca-certificates.crt -untrusted fullchain.pem cert.pem # openssl verify -show_chain -trusted /etc/ssl/certs/ca-certificates.crt -untrusted fullchain1.pem cert.pem # openssl verify -show_chain -partial_chain -trusted fullchain2.pem -untrusted fullchain1.pem cert.pem # (*) cert.pem: OK Chain: depth=0: CN = $FQHN (untrusted) depth=1: C = US, O = Let's Encrypt, CN = R3 (untrusted) depth=2: C = US, O = Internet Security Research Group, CN = ISRG Root X1 (Using `-trusted` implies `-no-CAfile -no-CApath -no-CAstore` and thus disables all other default sources for trusted CAs making the specified file the sole source.) See <https://www.openssl.org/docs/manmaster/man1/openssl-verification-options.html> for a full description of OpenSSL's verification process. (*): Please be aware that the last certificate provided by LetsEncrypt is an intermediate certificate cross-signed from IdenTrust for compatibility with older Android devices: <https://letsencrypt.org/2020/12/21/extending-android-compatibility.html> # openssl x509 -noout -subject -issuer -in fullchain2.pem subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1 issuer=O = Digital Signature Trust Co., CN = DST Root CA X3 That IdenTrutst CA is expired and no longer included with `ca-certificates` on Debian, but still trusted by older Android devices. But as the new "ISRG Root X1" is already included with `ca-certificates`, verification can stop there early: # openssl verify -show_chain -trusted /etc/ssl/certs/ISRG_Root_X1.pem -untrusted fullchain1.pem fullchain0.pem cert.pem: OK depth=0: CN = birdy.pmhahn.de (untrusted) depth=1: C = US, O = Let's Encrypt, CN = R3 (untrusted) depth=2: C = US, O = Internet Security Research Group, CN = ISRG Root X1
*** Bug 51787 has been marked as a duplicate of this bug. ***
Our "update from very old UCS"-tests where failing when SpamAssassin was installed, which tried to download its updates rules from https://spamassassin.apache.org/, which uses a certificate from Let's encrypt (Bug #53751). The SA-certificate-chain still contains the LE-cross-sign-certificate to "DST Root CA X3", which is expired by now, but may still be included in /etc/ssl/certs/ca-certificates.crt. # openssl x509 -noout -subject -issuer -startdate -enddate -in /etc/ssl/certs/2e5ac55d.0 subject=O = Digital Signature Trust Co., CN = DST Root CA X3 issuer=O = Digital Signature Trust Co., CN = DST Root CA X3 notBefore=Sep 30 21:12:19 2000 GMT notAfter=Sep 30 14:01:15 2021 GMT Many (older) implementations — OpenSSL included — will still follow that chain, find the expired certificate, and reports that as a failure. Only newer implementations do backtracking in case of expired certificates and will then directly used the newer "ISRG Root X1" certificate. The correct fix is to *remove expired CAs* from the bundle: > [ -f /etc/ca-certificates.conf/etc/ca-certificates.conf ] && > sed -i -e 's=^mozilla/DST_Root_CA_X3.crt=!&=' /etc/ca-certificates.conf && > update-ca-certificates || > true You can also do it via DEBCONF and `dpkg-reconfigure ca-certificates`, but that requires more work. AFAIK `update-ca-certificates` has no option to automatically remove expired CAs. Instead the "official Debian way" is to release an updated version of "ca-certificate", which removes the expired CA. `update-ca-certificates` will be triggered by `postinst` and will then remove the no-longer shipped certificates. But there never was an update after 2020-06 for Debian-9-Stretch (UCS 4.4), so the expired CA is still included: <https://packages.debian.org/search?keywords=ca-certificates&searchon=sourcenames&exact=1&suite=all§ion=all>. Even the current version in Debian-11-Bullseye "20210119" still contains `/usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt`.