Bug 52517 - Let's encrypt changes intermediate certificate
Let's encrypt changes intermediate certificate
Status: RESOLVED FIXED
Product: UCS
Classification: Unclassified
Component: Let's Encrypt
UCS 4.4
All Linux
: P5 critical (vote)
: UCS 4.4-7
Assigned To: Erik Damrose
UCS maintainers
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-12-18 16:04 CET by Daniel Krüger
Modified: 2021-01-21 17:59 CET (History)
8 users (show)

See Also:
What kind of report is it?: Bug Report
What type of bug is this?: 5: Major Usability: Impairs usability in key scenarios
Who will be affected by this bug?: 3: Will affect average number of installed domains
How will those affected feel about the bug?: 3: A User would likely not purchase the product
User Pain: 0.257
Enterprise Customer affected?: Yes
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number: 2021010521000624
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Krüger 2020-12-18 16:04:55 CET
According to https://letsencrypt.org/2020/11/06/own-two-feet.html and https://twitter.com/letsencrypt/status/1334568843927228418, Let's encrypt started on 3rd of December 2020 to issues certificates with its new R3 certificate.

I needed to replace /etc/univention/letsencrypt/intermediate.pem on my UCS server with https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem. Otherwise some client programs complained about there inability to trust the certificate. Among those programs was the Nextcloud client on Ubuntu.

Even "openssl s_client -connect <myserver>" complained with the following lines:
verify error:num=20:unable to get local issuer certificate
verify error:num=21:unable to verify the first certificate
Comment 1 Daniel Tröder univentionstaff 2021-01-02 13:13:22 CET
Workaround until fixed:

$ cd /etc/univention/letsencrypt
$ wget https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem
$ ucr set apache2/ssl/certificatechain=/etc/univention/letsencrypt/lets-encrypt-r3-cross-signed.pem
$ service apache2 restart
Comment 3 Erik Damrose univentionstaff 2021-01-06 09:49:16 CET
*** Bug 52546 has been marked as a duplicate of this bug. ***
Comment 4 Christian Castens univentionstaff 2021-01-15 16:21:22 CET
The new R3 intermediate certificate has been added to the app.

Successful build
Package: univention-letsencrypt
Version: 1.2.2-9A~4.4.0.202101151612
Branch: ucs_4.4-0
Scope: letsencrypt

commits (4.3):
b059238a65d57e9e104b255bd6800a5411b0c40a (changelog, new certificate)
b20c8589150db8cba674df10a19b5a36882bf8ec (set permissions for certificate)
Comment 5 Felix Botner univentionstaff 2021-01-18 17:35:49 CET
(1)
openssl verify still fails

-> openssl verify /etc/univention/letsencrypt/signed_chain.crt
CN = le-fbtest1.dev-univention.de
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/univention/letsencrypt/signed_chain.crt: verification failed

also after /usr/share/univention-letsencrypt/refresh-cert-cron, but the new signed_chain.crt contains the new intermediate.pem, so mabye openssl verify is not the right tool to check?

(2)
We now ship the intermediate.pem with the package. But this certificate expires in Sep 2021, and we would have to update the app again. Maybe the "old" way is better after all,  wget the correct intermediate.pem in setup-letsencrypt?
Comment 6 Daniel Krüger 2021-01-18 19:57:18 CET
According to https://letsencrypt.org/2020/12/21/extending-android-compatibility.html, Let's encrypt will/has changed the intermediate certificates again. This latest change will make the Let's encrypt certifcates work again on Android prior to 7.1.1.

I would appreciate if UCS could support this change, so we don't lose contact to older Android clients.
Comment 7 Julia Bremer univentionstaff 2021-01-19 23:46:18 CET
Successful build
Package: univention-letsencrypt
Version: 1.2.2-10A~4.4.0.202101192324
Branch: ucs_4.4-0
Scope: letsencrypt

3913ed3 Bug #52517: version bump
ac26daa Revert "Bug #52517: permissions of intermediate cert are always set during installation"

3913ed3 (HEAD -> 4.3, origin/HEAD, origin/4.3) Bug #52517: version bump
296458b get new cert
ac26daa Revert "Bug #52517: permissions of intermediate cert are always set during installation"

I reverted the prior changes. 
I changed the link in the setup-letsencrypt script.
I adjusted the postinst, so that setup-letsencrypt is run after this update and 
the new cert is download. The signed_chain.cert is refreshed afterwards.
Comment 8 Erik Damrose univentionstaff 2021-01-21 17:13:15 CET
(In reply to Daniel Krüger from comment #0)
> I needed to replace /etc/univention/letsencrypt/intermediate.pem on my UCS
> server with https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem.
> Otherwise some client programs complained about there inability to trust the
> certificate. Among those programs was the Nextcloud client on Ubuntu.

As i see it, the intermediate cert is only downloaded to make it known to UCS internally, this would not fix any client issues that are unable to verify requests to UCS. That cert was never offered to clients in a default app installation.

Our acme_tiny client talks to the letsencrypt (LE) apiv2 endpoint, so certs issued already contain the complete chain, including the R3 intermediate:
# openssl crl2pkcs7 -nocrl -certfile /etc/univention/letsencrypt/signed_chain.crt | openssl pkcs7 -print_certs -noout
subject=/CN=my.valid.testdomain
issuer=/C=US/O=Let's Encrypt/CN=R3

subject=/C=US/O=Let's Encrypt/CN=R3
issuer=/O=Digital Signature Trust Co./CN=DST Root CA X3

> Even "openssl s_client -connect <myserver>" complained with the following
> lines:

The UCS letsencrypt app creates a separate VirtualHost entry for the LE site. The openssl command above misses the SNI parameter, so the wrong site is requested. Use "openssl s_client -connect <myserver> -servername <myserver>" to query the correct VirtualHost.

In our tests we could not reproduce any error with fresh LE certificates. We use the LE app on several of our own servers and no errors were reported internally. -> We will resolve this as RESOLVED WORKSFORME if no new information comes up.
Note: Issues with the UMC diagnostic module complaining about letsencrypt are tracked at bug 52546.

One more note: Letsencrypt changed their cert cross signing once more, the change should be live now or in the near future according to https://letsencrypt.org/2020/12/21/extending-android-compatibility.html
This should improve certificate verification on older devices.

Recreating the certificate once could help.

todo: revert current changes to package
Comment 9 Erik Damrose univentionstaff 2021-01-21 17:16:37 CET
Ticket 2021010521000544 seems completely unrelated to this bug, removed
I think 2021010521000624 is unrelated but am not really sure
Comment 10 Julia Bremer univentionstaff 2021-01-21 17:59:49 CET
Reverted

Package: univention-letsencrypt
Version: 1.2.2-14A~4.4.0.202101211759
Branch: ucs_4.4-0
Scope: letsencrypt

454f4ba (HEAD -> 4.3, origin/HEAD, origin/4.3) Bug #52517: revert changes. the diagnostic check reported